Nov 03 19:54:10.748873 ip-10-0-135-70 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 19:54:10.748884 ip-10-0-135-70 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 19:54:10.748891 ip-10-0-135-70 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 19:54:10.749136 ip-10-0-135-70 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 19:54:20.979977 ip-10-0-135-70 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 19:54:20.979997 ip-10-0-135-70 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6ee72659cb2742c1bb1cd4a23aef1e75 -- Nov 03 19:56:27.287432 ip-10-0-135-70 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:56:27.669905 ip-10-0-135-70 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:27.669905 ip-10-0-135-70 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:56:27.669905 ip-10-0-135-70 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:27.669905 ip-10-0-135-70 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:27.676743 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676645 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.676743 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676742 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.676743 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676746 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.676743 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676749 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676753 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676756 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676759 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676761 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676764 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676767 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676770 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676773 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676777 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676780 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676783 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676786 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676788 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676791 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676793 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676796 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676799 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676802 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.676872 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676804 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676807 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676809 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676812 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676818 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676824 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676827 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676830 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676833 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676838 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676840 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676843 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676845 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676849 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676851 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676854 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676857 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676860 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.677325 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676862 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676865 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676868 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676870 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676873 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676876 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676879 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676883 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676886 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676889 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676892 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676894 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676897 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676900 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676902 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676905 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676908 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676911 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676914 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676917 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.677803 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676920 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676922 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676925 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676927 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676930 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676934 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676938 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676943 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676946 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676949 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676952 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676957 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676959 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676962 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676965 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676968 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676971 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676974 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676976 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.678288 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676979 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676981 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676984 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676987 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676989 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676992 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.676996 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677000 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677003 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677006 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677009 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677012 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677014 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677022 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677025 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677028 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677030 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677033 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677035 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.678782 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677040 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677043 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677046 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677048 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677051 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677054 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677973 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677982 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677986 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677990 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677993 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.677997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678000 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678003 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678006 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678010 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678014 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678018 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.679281 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678020 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678023 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678026 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678029 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678031 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678035 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678037 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678040 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678043 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678046 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678049 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678052 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678054 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678057 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678060 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678063 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678065 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678068 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678071 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678073 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.679725 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678076 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678079 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678082 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678087 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678090 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678093 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678095 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678098 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678101 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678117 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678120 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678123 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678125 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678129 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678133 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678137 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678140 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678144 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678147 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.680211 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678150 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678153 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678155 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678159 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678162 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678165 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678167 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678170 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678173 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678176 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678179 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678181 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678184 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678186 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678189 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678192 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678197 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678199 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678202 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678205 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.680661 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678210 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678213 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678215 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678218 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678221 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678223 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678226 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678228 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678231 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678233 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678236 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678238 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678241 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678244 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678246 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678256 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678259 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678262 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.681155 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678265 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678267 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678270 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678273 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678275 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678278 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678281 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678284 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678286 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678290 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678293 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678296 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678299 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678302 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678306 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678308 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678828 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678835 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678838 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678841 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.681597 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678844 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678850 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678853 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678857 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678861 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678864 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678867 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678870 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678873 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678879 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678881 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678884 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678887 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678890 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678892 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678895 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678898 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.682070 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678900 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678903 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678906 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678909 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678913 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678915 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678918 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678920 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678923 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678926 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678928 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678931 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678933 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678936 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678938 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678941 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678944 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678947 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678949 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678952 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.682541 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678955 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678957 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678960 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678963 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678965 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678968 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678970 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678973 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678976 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678978 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678981 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678984 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678986 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678989 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678992 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678995 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.678997 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679000 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679002 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.683007 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679005 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679007 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679010 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679013 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679015 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679018 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679021 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679023 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679026 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679028 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679031 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679033 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679036 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679038 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679041 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679043 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679045 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679049 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679052 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679054 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.683613 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679057 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679062 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679064 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679067 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679069 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679072 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679075 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679078 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679080 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679083 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679085 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679087 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679090 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679094 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679097 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679099 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679119 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679122 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679124 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.684090 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679127 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679130 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679133 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.679135 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680122 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680138 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680146 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680151 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680155 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680159 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680164 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680170 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680173 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680177 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680180 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680184 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680187 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680190 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680193 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680197 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680200 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680203 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680207 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:56:27.684586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680210 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680213 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680217 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680221 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680225 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680228 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680234 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680238 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680241 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680244 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680247 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680250 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680254 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680258 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680262 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680265 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680268 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680271 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680275 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680280 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680285 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680288 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680292 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680295 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:56:27.685145 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680300 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680303 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680307 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680310 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680313 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680317 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680320 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680323 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680326 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680329 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680333 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680337 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680340 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680344 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680348 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680352 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680355 2579 flags.go:64] FLAG: --help="false" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680359 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680362 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680365 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680368 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680372 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680375 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:56:27.685747 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680378 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680381 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680385 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680387 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680391 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680395 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680399 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680402 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680405 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680408 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680411 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680414 2579 flags.go:64] FLAG: --lock-file="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680417 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680420 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680424 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680429 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680433 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680436 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680439 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680442 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680446 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680449 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680452 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680458 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680462 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:56:27.686309 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680466 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680469 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680472 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680475 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680479 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680482 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680485 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680488 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680496 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680499 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680502 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680505 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680510 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680513 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680516 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680519 2579 flags.go:64] FLAG: --port="10250" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680523 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680526 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08639579d593024cd" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680529 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680532 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680535 2579 flags.go:64] FLAG: --register-node="true" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680538 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680542 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680546 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680549 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680552 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:56:27.686941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680556 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680559 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680562 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680565 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680569 2579 flags.go:64] FLAG: --runonce="false" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680572 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680576 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680579 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680582 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680585 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680588 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680592 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680595 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680598 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680601 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680604 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680607 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680610 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680614 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680617 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680623 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680626 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680630 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680634 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680637 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:56:27.687683 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680640 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680643 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680646 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680649 2579 flags.go:64] FLAG: --v="2" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680655 2579 flags.go:64] FLAG: --version="false" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680659 2579 flags.go:64] FLAG: --vmodule="" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680664 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.680667 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680805 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680809 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680812 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680816 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680818 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680821 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680823 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680826 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680829 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680831 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680837 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680842 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.688292 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680845 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680850 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680853 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680856 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680859 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680861 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680864 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680867 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680871 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680875 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680878 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680880 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680883 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680886 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680888 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680891 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680894 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680896 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.688824 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680899 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680902 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680904 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680907 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680910 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680915 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680917 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680920 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680923 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680925 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680928 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680931 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680933 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680935 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680938 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680941 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680944 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680946 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680949 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.689298 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680951 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680954 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680958 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680960 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680963 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680966 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680968 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680971 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680973 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680976 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680980 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680984 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680987 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680990 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680992 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680995 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.680998 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681000 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.689769 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681003 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681008 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681011 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681014 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681017 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681019 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681022 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681024 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681027 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681029 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681032 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681035 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681037 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681040 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681042 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681045 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681048 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681051 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681053 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.690209 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681056 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681058 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681061 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681063 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681066 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681068 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681071 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681074 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681076 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681079 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681082 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681085 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681087 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681090 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681092 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.690691 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.681133 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681170 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681175 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681178 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681181 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681183 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681186 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681189 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681191 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681194 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681197 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681200 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681202 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681205 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681214 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681217 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681220 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681223 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681225 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.691054 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681228 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681230 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681233 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681237 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681240 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681243 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681248 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681251 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681257 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681259 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681262 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681265 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681268 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681270 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681273 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681276 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681278 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681281 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.691557 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681283 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681289 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681291 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681294 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681297 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681299 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681302 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681307 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681310 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681312 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681315 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681318 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681320 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681323 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681326 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681328 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681331 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681334 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.692002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681338 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681340 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681343 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681346 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681349 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681352 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681355 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681358 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681360 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681363 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681365 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681368 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681371 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681373 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681376 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681378 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681381 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681384 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681386 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681389 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.692547 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681391 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681396 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681399 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681402 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681404 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681407 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681410 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681412 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681415 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681418 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681420 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681424 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681429 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681432 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681435 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681437 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681440 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681442 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681445 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.693048 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681448 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681450 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681453 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681456 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681458 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681461 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681463 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681466 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681469 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.681471 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.688757 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.688776 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.688793 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.688797 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688860 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688866 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688869 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688872 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.693522 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688879 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688882 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688885 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688888 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688891 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688894 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688896 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688899 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688902 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688905 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688908 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688911 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688914 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688917 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688919 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688922 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688925 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688928 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688930 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688933 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.693974 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688936 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688938 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688941 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688943 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688948 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688951 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688954 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688956 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688959 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688961 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688964 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688967 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688969 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688972 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688977 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688979 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688982 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688984 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.694465 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688987 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688990 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688993 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688996 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.688998 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689001 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689003 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689006 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689009 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689011 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689014 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689018 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689022 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689026 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689029 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689031 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689034 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689037 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689040 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.694921 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689043 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689046 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689049 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689051 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689054 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689062 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689066 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689070 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689073 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689076 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689078 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689082 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689084 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689087 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689090 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689095 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.695492 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689097 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689100 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689115 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689118 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689121 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689123 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689126 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689129 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689131 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689134 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689136 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689140 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689142 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689146 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689148 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689151 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689154 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689156 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689159 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.696230 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689161 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689164 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689167 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689170 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689172 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.689201 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689344 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689349 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689351 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689354 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689357 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689360 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689363 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689366 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689368 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689371 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.696904 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689374 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689376 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689379 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689381 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689383 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689386 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689389 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689392 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689395 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689397 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689400 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689404 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689407 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689409 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689412 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689414 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689417 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689419 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689422 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689425 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.697412 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689427 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689430 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689433 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689435 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689438 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689440 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689443 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689446 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689448 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689451 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689454 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689456 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689458 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689461 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689464 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689466 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689469 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689471 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689474 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.697915 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689478 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689482 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689486 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689489 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689492 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689495 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689498 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689501 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689503 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689506 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689508 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689512 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689516 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689519 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689522 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689525 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689528 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689531 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689533 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.698378 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689536 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689539 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689542 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689545 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689547 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689550 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689552 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689555 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689557 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689565 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689568 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689570 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689573 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689576 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689579 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689581 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689584 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689587 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.698865 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689589 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689592 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689595 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689597 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689602 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689605 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689607 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689610 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689612 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689615 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689617 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689620 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689622 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689625 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689627 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689630 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.699360 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689633 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.689658 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689688 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689692 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689695 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689699 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689702 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689705 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689707 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689710 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689713 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689715 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689719 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689722 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689725 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689728 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:27.699764 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689731 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689734 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689737 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689739 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689742 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689745 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689748 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689750 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689753 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689756 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689758 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689761 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689764 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689766 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689769 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689772 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689775 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689782 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:27.700227 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689785 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689787 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689790 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689792 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689795 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689798 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689800 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689803 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689805 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689809 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689811 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689814 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689816 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689820 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689822 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689825 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689827 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689830 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689832 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689835 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:27.700676 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689837 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689840 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689843 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689845 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689848 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689850 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689853 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689856 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689858 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689861 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689864 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689867 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689870 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689872 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689877 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689880 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689885 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689888 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:27.701267 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689891 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689894 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689897 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689900 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689903 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689905 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689908 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689912 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689914 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689918 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689922 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689925 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689927 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689930 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689932 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689935 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689937 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689940 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689943 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:27.701745 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689945 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689948 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689951 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689953 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689957 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689960 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689963 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689966 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689969 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689971 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689974 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689976 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689979 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689981 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:27.689984 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.690042 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.692185 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.693343 2579 server.go:1014] "Starting client certificate rotation" Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.693849 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:27.702202 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.693893 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:27.702657 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.695663 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:27.702657 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.697418 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:27.709698 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.709676 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:56:27.713121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.713095 2579 log.go:25] "Validated CRI v1 image API" Nov 03 19:56:27.714001 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.713977 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:56:27.716648 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.716622 2579 fs.go:135] Filesystem UUIDs: map[473aad34-b4bc-48e8-bd6b-9d19cf305d57:/dev/nvme0n1p4 4f429464-8a0b-44ee-b611-69c67c790439:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 19:56:27.716724 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.716647 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:56:27.721760 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.721627 2579 manager.go:217] Machine: {Timestamp:2026-11-03 19:56:27.72039579 +0000 UTC m=+0.306924271 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3034363 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b95a672c39aab484d22307f0a736 SystemUUID:ec28b95a-672c-39aa-b484-d22307f0a736 BootID:6ee72659-cb27-42c1-bb1c-d4a23aef1e75 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:57:1e:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:57:1e:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:84:0c:32:f2:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:56:27.721760 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.721750 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:56:27.721939 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.721864 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:56:27.724694 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724660 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:56:27.724865 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724695 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:56:27.724942 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724870 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:56:27.724942 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724883 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:56:27.725037 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724941 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:27.725037 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.724968 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:27.725759 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.725745 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:56:27.729494 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729477 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:56:27.729638 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729627 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:56:27.729692 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729667 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:56:27.729692 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729684 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:56:27.729780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729700 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:56:27.729780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729716 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:56:27.729780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.729729 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:56:27.730121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.730092 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:27.732824 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.732796 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:56:27.734016 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.733994 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:27.737543 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.737526 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:27.738420 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738407 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:56:27.738461 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738438 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:56:27.738618 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738608 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:56:27.738649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738623 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:56:27.738649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738629 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:56:27.738649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738635 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:56:27.738649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738641 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:56:27.738649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.738647 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:56:27.739165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739156 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:56:27.739165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739166 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:56:27.739225 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739173 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:56:27.739225 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739179 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:56:27.739225 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739188 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:56:27.739225 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739198 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:56:27.739778 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739769 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:56:27.739809 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.739779 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:56:27.745964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.745940 2579 server.go:1263] "Started kubelet" Nov 03 19:56:27.746067 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.746018 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:56:27.746162 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.746095 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:56:27.746234 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.746182 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:56:27.746790 ip-10-0-135-70 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:56:27.747303 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.747281 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:56:27.748228 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.748214 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:56:27.748415 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.748351 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-70.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:56:27.752240 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.752214 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:56:27.752350 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.752326 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-70.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:56:27.753077 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.752179 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-70.us-east-2.compute.internal.18e4a132017fe7ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-70.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-70.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:27.745896398 +0000 UTC m=+0.332424879,LastTimestamp:2026-11-03 19:56:27.745896398 +0000 UTC m=+0.332424879,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-70.us-east-2.compute.internal,}" Nov 03 19:56:27.753077 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.753042 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:56:27.753319 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.753305 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:27.753428 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.753410 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cm9hq" Nov 03 19:56:27.754169 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.754150 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:56:27.754673 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.754654 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:27.755024 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755004 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:56:27.755140 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755047 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:56:27.755140 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755063 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:56:27.755230 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755205 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:56:27.755230 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755214 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:56:27.755327 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755268 2579 factory.go:55] Registering systemd factory Nov 03 19:56:27.755327 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755278 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:56:27.755573 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755561 2579 factory.go:153] Registering CRI-O factory Nov 03 19:56:27.755573 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755572 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 19:56:27.755666 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755632 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:56:27.755666 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755651 2579 factory.go:103] Registering Raw factory Nov 03 19:56:27.755666 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.755667 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 19:56:27.756551 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.756528 2579 manager.go:319] Starting recovery of all containers Nov 03 19:56:27.764410 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.764384 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cm9hq" Nov 03 19:56:27.765267 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.765240 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-70.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:56:27.767186 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.767164 2579 manager.go:324] Recovery completed Nov 03 19:56:27.768451 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.768420 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:56:27.771938 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.771919 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:27.774030 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.774007 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:27.774121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.774048 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:27.774121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.774063 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:27.775260 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.775245 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:56:27.775260 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.775259 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:56:27.775381 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.775244 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:27.775381 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.775278 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:56:27.776058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.776043 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:27.776127 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.776070 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:27.776127 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.776081 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:27.777212 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.777196 2579 policy_none.go:50] "Start" Nov 03 19:56:27.777212 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.777214 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:56:27.777313 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.777228 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:56:27.779259 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.779245 2579 policy_none.go:44] "Start" Nov 03 19:56:27.806803 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.806762 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:56:27.810895 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.810874 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 19:56:27.810977 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.810954 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:56:27.810977 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.810963 2579 server.go:87] "Starting device plugin registration server" Nov 03 19:56:27.811238 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.811225 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:56:27.811321 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.811237 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:56:27.811381 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.811336 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:56:27.811429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.811410 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:56:27.811429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.811419 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:56:27.811943 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.811924 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:56:27.812047 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.811975 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:27.895917 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.895877 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:56:27.898071 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.897100 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:56:27.898071 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.897166 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:56:27.898071 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.897186 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:56:27.898071 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.897223 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 19:56:27.901172 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.901152 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:27.912228 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.912207 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:27.913188 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.913167 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:27.913280 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.913204 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:27.913280 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.913218 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:27.913280 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.913243 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:27.922031 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.921983 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:27.922031 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.922011 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-70.us-east-2.compute.internal\": node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:27.937192 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:27.937170 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:27.997564 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.997361 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal"] Nov 03 19:56:27.997718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.997605 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:27.998626 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.998608 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:27.998690 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.998640 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:27.998690 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:27.998651 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:28.000892 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.000878 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:28.001065 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.001148 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001086 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:28.001706 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001681 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:28.001804 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001715 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:28.001804 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001725 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:28.001804 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001732 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:28.001804 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001744 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:28.001804 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.001754 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:28.003865 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.003848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.003948 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.003874 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:28.004675 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.004660 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:28.004740 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.004687 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:28.004740 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.004701 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:28.036044 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.036019 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.037616 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.037595 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.038482 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.038466 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.056808 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.056780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.056808 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.056809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.056955 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.056827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.138581 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.138527 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.157976 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.157943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.158063 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.157980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.158063 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.158000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.158063 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.158053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.158224 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.158057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.158224 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.158129 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.239457 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.239385 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.338941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.338910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.339970 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.339953 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.342153 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.342137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.440067 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.440024 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.545451 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.545378 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.646015 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.645968 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.682417 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.682391 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:28.693114 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.693087 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:56:28.693286 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.693262 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:28.693286 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.693277 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:28.746211 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:28.746172 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:56:28.774861 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.774804 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:28 +0000 UTC" deadline="2028-05-23 11:44:38.922429809 +0000 UTC" Nov 03 19:56:28.774861 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.774854 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13599h48m10.147579021s" Nov 03 19:56:28.805793 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.805723 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:28.855286 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.855257 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.866813 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.866791 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:28.868437 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.868421 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:56:28.880363 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:28.880334 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:28.997894 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:28.997852 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd161bc542803fe55ee71704a451b4d4d.slice/crio-947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a WatchSource:0}: Error finding container 947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a: Status 404 returned error can't find the container with id 947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a Nov 03 19:56:29.003720 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:29.003698 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44458e05214beace7a3b673d5f048430.slice/crio-c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0 WatchSource:0}: Error finding container c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0: Status 404 returned error can't find the container with id c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0 Nov 03 19:56:29.005367 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.005352 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:56:29.114494 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.114408 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:29.731204 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.731171 2579 apiserver.go:52] "Watching apiserver" Nov 03 19:56:29.739274 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.739240 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:56:29.739634 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.739613 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lb594","openshift-multus/multus-additional-cni-plugins-njwcg","openshift-network-diagnostics/network-check-target-r97c6","openshift-network-operator/iptables-alerter-tnxjb","openshift-ovn-kubernetes/ovnkube-node-l9nr5","kube-system/konnectivity-agent-8wq99","openshift-cluster-node-tuning-operator/tuned-49krk","openshift-dns/node-resolver-v2tvg","openshift-image-registry/node-ca-87bj5","openshift-multus/multus-ndfvw","openshift-multus/network-metrics-daemon-psmsw","kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal"] Nov 03 19:56:29.742390 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.742355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.744778 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.744743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.754198 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:29.754325 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.754277 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:29.754466 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754445 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:56:29.754601 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754585 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.754652 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.754736 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754722 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:56:29.754839 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754811 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:29.754954 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754864 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.754954 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.754933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.756230 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.755209 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:56:29.756230 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.755886 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.756404 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.756310 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.757565 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.757542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.758827 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.758790 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:56:29.760659 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.760641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.763131 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.763102 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:56:29.763219 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.763157 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:56:29.764599 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.764578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.764693 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.764654 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:29.767108 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767108 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767108 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767337 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98k5c\" (UniqueName: \"kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767337 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767337 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767337 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.767337 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.767963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.767998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnwr\" (UniqueName: \"kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mqs\" (UniqueName: \"kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.768816 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.768876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.769050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.769081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.769106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.769385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.769152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgr8h\" (UniqueName: \"kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.771204 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.770739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.771663 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.771578 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:29.773606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.773362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.773606 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.773430 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:29.775895 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.775875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.790845 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.790820 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n86xh" Nov 03 19:56:29.803780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.803721 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n86xh" Nov 03 19:56:29.856684 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.856648 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:56:29.856927 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.856904 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.857001 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.856922 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.857001 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.856994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.857104 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857041 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.857155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857145 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.857278 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857263 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.857345 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857319 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:56:29.857404 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857377 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:56:29.857404 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857385 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:56:29.857506 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857468 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.857506 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857481 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:56:29.857605 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.857562 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:56:29.859129 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859103 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:56:29.859262 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859161 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:56:29.859262 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859210 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:56:29.859262 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859241 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:56:29.859262 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859253 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859280 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859303 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859333 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859371 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859382 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859397 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859451 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:56:29.859470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859452 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:56:29.859868 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.859491 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:56:29.869629 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.869792 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt22\" (UniqueName: \"kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxjr\" (UniqueName: \"kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869970 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.869979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.870165 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870185 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870744 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.870973 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870979 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.870989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcl46\" (UniqueName: \"kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871381 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.871528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.872161 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.871998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnwr\" (UniqueName: \"kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mqs\" (UniqueName: \"kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.873010 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlc9\" (UniqueName: \"kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgr8h\" (UniqueName: \"kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.872949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzfzx\" (UniqueName: \"kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98k5c\" (UniqueName: \"kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873591 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873744 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873824 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.873859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.874862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874354 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.874862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.874862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.875118 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.875118 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.874961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.875118 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.875320 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875147 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.875320 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.875867 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.875867 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.875867 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.876736 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.875898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.876736 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.876276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.876736 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.876537 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.876887 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.876846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.877927 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.877902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.879814 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.879767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.880239 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.880219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.886906 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.886880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:29.887581 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.887557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98k5c\" (UniqueName: \"kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:29.888760 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.888446 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:29.888760 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.888470 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:29.888760 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.888485 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:29.888760 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.888570 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.388537743 +0000 UTC m=+2.975066233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:29.889623 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.889590 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mqs\" (UniqueName: \"kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:29.890888 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.890855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgr8h\" (UniqueName: \"kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:29.892909 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.892882 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnwr\" (UniqueName: \"kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:29.902338 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.902278 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerStarted","Data":"c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0"} Nov 03 19:56:29.903830 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.903775 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"d161bc542803fe55ee71704a451b4d4d","Type":"ContainerStarted","Data":"947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a"} Nov 03 19:56:29.973702 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.973702 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973850 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlc9\" (UniqueName: \"kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.973964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.973998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974063 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzfzx\" (UniqueName: \"kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt22\" (UniqueName: \"kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.974364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxjr\" (UniqueName: \"kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcl46\" (UniqueName: \"kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.974889 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.975707 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.975707 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975200 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.975707 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.975707 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.974277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.975707 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.975960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.975960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.975960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.975960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.975865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.976623 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.976596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.976761 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.976738 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.976854 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.976837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.976930 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.976894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.976993 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.976977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.977057 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.977057 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.977031 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:29.977162 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.977162 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:29.977162 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.977147 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.477127343 +0000 UTC m=+3.063655831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:29.977346 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.977346 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977248 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.977346 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.977346 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.977314 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:29.977529 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:29.977374 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.477352017 +0000 UTC m=+3.063880486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:29.977529 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977389 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.977529 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:29.977780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.977752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.978031 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.978010 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.978859 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.978839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.988922 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.988835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzfzx\" (UniqueName: \"kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:29.988922 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.988840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcl46\" (UniqueName: \"kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:29.989136 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.989022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxjr\" (UniqueName: \"kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:29.990334 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.990288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlc9\" (UniqueName: \"kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:56:29.992454 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:29.992420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt22\" (UniqueName: \"kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:30.068377 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.068340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:56:30.071172 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.071138 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:30.079956 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.079918 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cbdc4a_e9f3_4489_bd72_341746fdc0c9.slice/crio-791a8438794d0a215de81e8c4a475947ac723d3495241189c20912f540b39666 WatchSource:0}: Error finding container 791a8438794d0a215de81e8c4a475947ac723d3495241189c20912f540b39666: Status 404 returned error can't find the container with id 791a8438794d0a215de81e8c4a475947ac723d3495241189c20912f540b39666 Nov 03 19:56:30.081075 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.080866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:56:30.087263 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.087232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:56:30.090975 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.090949 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode458f2d3_a88a_44b2_8bd9_57bbb097bfa8.slice/crio-b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818 WatchSource:0}: Error finding container b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818: Status 404 returned error can't find the container with id b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818 Nov 03 19:56:30.097217 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.097180 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945ef139_c1e8_4154_b869_f18adcc62d8a.slice/crio-8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445 WatchSource:0}: Error finding container 8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445: Status 404 returned error can't find the container with id 8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445 Nov 03 19:56:30.111587 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.111550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:30.113825 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.113780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:30.115552 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.115529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:56:30.125293 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.125261 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216dd34c_6ab8_4044_97d2_d2ed537c6153.slice/crio-9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e WatchSource:0}: Error finding container 9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e: Status 404 returned error can't find the container with id 9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e Nov 03 19:56:30.125874 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.125852 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06728ec3_24de_459c_b966_8b88db3bc248.slice/crio-cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61 WatchSource:0}: Error finding container cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61: Status 404 returned error can't find the container with id cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61 Nov 03 19:56:30.131538 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.131505 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60f9fad_b8c0_4d6e_b755_c2bda3f2d014.slice/crio-a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7 WatchSource:0}: Error finding container a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7: Status 404 returned error can't find the container with id a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7 Nov 03 19:56:30.138355 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.138333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndfvw" Nov 03 19:56:30.146211 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.146175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:56:30.149859 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.149822 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c657c7_cf56_4ca8_b035_fd7c55fc2480.slice/crio-cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea WatchSource:0}: Error finding container cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea: Status 404 returned error can't find the container with id cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea Nov 03 19:56:30.153674 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.153653 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:56:30.157442 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.157407 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf52cd_afe4_46cb_b9b3_148d5445ca6e.slice/crio-c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4 WatchSource:0}: Error finding container c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4: Status 404 returned error can't find the container with id c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4 Nov 03 19:56:30.167702 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:56:30.167665 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3371a194_9a9e_4a3a_9af3_48d86152b1dc.slice/crio-f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097 WatchSource:0}: Error finding container f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097: Status 404 returned error can't find the container with id f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097 Nov 03 19:56:30.480706 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.480611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:30.480706 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.480687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.480740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480792 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480866 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480875 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:31.480853948 +0000 UTC m=+4.067382432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480884 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480895 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:30.480941 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480946 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:31.48093108 +0000 UTC m=+4.067459566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:30.481359 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.480996 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:30.481359 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.481033 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:31.4810214 +0000 UTC m=+4.067549885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:30.546482 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.546452 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:30.807388 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.807252 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:30 +0000 UTC" deadline="2028-05-09 11:44:30.781993656 +0000 UTC" Nov 03 19:56:30.807388 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.807327 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13263h47m59.974671332s" Nov 03 19:56:30.897540 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.897466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:30.897540 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.897509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:30.897540 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.897534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:30.897832 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.897665 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:30.898258 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.898090 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:30.898258 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:30.898207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:30.912835 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.912801 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-87bj5" event={"ID":"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014","Type":"ContainerStarted","Data":"a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7"} Nov 03 19:56:30.914568 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.914538 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnxjb" event={"ID":"945ef139-c1e8-4154-b869-f18adcc62d8a","Type":"ContainerStarted","Data":"8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445"} Nov 03 19:56:30.916760 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.916718 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2tvg" event={"ID":"3371a194-9a9e-4a3a-9af3-48d86152b1dc","Type":"ContainerStarted","Data":"f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097"} Nov 03 19:56:30.920771 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.920739 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerStarted","Data":"c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4"} Nov 03 19:56:30.926893 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.926868 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndfvw" event={"ID":"32c657c7-cf56-4ca8-b035-fd7c55fc2480","Type":"ContainerStarted","Data":"cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea"} Nov 03 19:56:30.929168 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.929141 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61"} Nov 03 19:56:30.940257 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.940226 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wq99" event={"ID":"216dd34c-6ab8-4044-97d2-d2ed537c6153","Type":"ContainerStarted","Data":"9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e"} Nov 03 19:56:30.946955 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.946917 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" event={"ID":"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8","Type":"ContainerStarted","Data":"b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818"} Nov 03 19:56:30.956078 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:30.956052 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-49krk" event={"ID":"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9","Type":"ContainerStarted","Data":"791a8438794d0a215de81e8c4a475947ac723d3495241189c20912f540b39666"} Nov 03 19:56:31.489587 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.489502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:31.489587 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.489558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:31.489811 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.489596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:31.489811 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.489771 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:31.489910 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.489839 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.489819522 +0000 UTC m=+6.076347997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:31.490240 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490223 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:31.490326 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490244 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:31.490326 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490254 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:31.490326 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490296 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.490275894 +0000 UTC m=+6.076804378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:31.490491 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490352 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:31.490491 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:31.490378 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:33.49037 +0000 UTC m=+6.076898483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:31.808075 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.807739 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:30 +0000 UTC" deadline="2028-08-14 11:17:03.871365498 +0000 UTC" Nov 03 19:56:31.808075 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.808023 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15591h20m32.063349677s" Nov 03 19:56:31.967083 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.966500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"d161bc542803fe55ee71704a451b4d4d","Type":"ContainerStarted","Data":"63aa235581154c71d68d03b3d700d67a82109a9bde65cd375988bbcc1cddac4b"} Nov 03 19:56:31.992721 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:31.992662 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" podStartSLOduration=2.9926425070000002 podStartE2EDuration="2.992642507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:31.991469249 +0000 UTC m=+4.577997744" watchObservedRunningTime="2026-11-03 19:56:31.992642507 +0000 UTC m=+4.579171005" Nov 03 19:56:32.897988 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:32.897964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:32.898358 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:32.898098 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:32.898358 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:32.898288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:32.898358 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:32.898296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:32.898471 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:32.898392 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:32.898507 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:32.898474 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:32.984649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:32.984065 2579 generic.go:356] "Generic (PLEG): container finished" podID="44458e05214beace7a3b673d5f048430" containerID="34a096f4cfa93c9576f7f046822e0132ef6896c923e9c566b7bf3eaf919e6cd3" exitCode=0 Nov 03 19:56:32.984649 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:32.984156 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerDied","Data":"34a096f4cfa93c9576f7f046822e0132ef6896c923e9c566b7bf3eaf919e6cd3"} Nov 03 19:56:33.505757 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:33.505658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:33.505757 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:33.505728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:33.505972 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:33.505779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:33.505972 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.505910 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:33.505972 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.505962 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:37.505948759 +0000 UTC m=+10.092477229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506303 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506330 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506334 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506346 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506401 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:37.506381758 +0000 UTC m=+10.092910243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:33.506568 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:33.506422 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:37.506411587 +0000 UTC m=+10.092940061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:33.990640 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:33.990532 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="d9fd02d8a7100347fd3d65a56aeaaf38cd17b03edb7a0ef36e7c90421693112b" exitCode=0 Nov 03 19:56:33.990640 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:33.990635 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"d9fd02d8a7100347fd3d65a56aeaaf38cd17b03edb7a0ef36e7c90421693112b"} Nov 03 19:56:34.002248 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:34.002221 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerStarted","Data":"eec22cdcda6c988e2748c66f56181755c4f73855d2f49e03c7bc73167ca4749c"} Nov 03 19:56:34.898366 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:34.898334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:34.898575 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:34.898486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:34.898840 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:34.898824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:34.898970 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:34.898913 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:34.898970 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:34.898954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:34.899077 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:34.899002 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:35.013292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:35.013254 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8wq99" event={"ID":"216dd34c-6ab8-4044-97d2-d2ed537c6153","Type":"ContainerStarted","Data":"a937b62368ec24df8a692aad39cb4f89afeafbc928df60d3ae08092d8abd13a4"} Nov 03 19:56:35.033128 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:35.032089 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" podStartSLOduration=6.032069071 podStartE2EDuration="6.032069071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:34.044312091 +0000 UTC m=+6.630840587" watchObservedRunningTime="2026-11-03 19:56:35.032069071 +0000 UTC m=+7.618597609" Nov 03 19:56:36.022702 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.022665 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-87bj5" event={"ID":"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014","Type":"ContainerStarted","Data":"2f733461ef0b77ae6dfb3b5e86056a64d6870568815f3e7b6ee8c7b0717c248a"} Nov 03 19:56:36.039843 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.039779 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8wq99" podStartSLOduration=3.899814815 podStartE2EDuration="8.039760676s" totalImagesPullingTime="4.139945861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:35.03235847 +0000 UTC m=+7.618886966" watchObservedRunningTime="2026-11-03 19:56:36.039760676 +0000 UTC m=+8.626289165" Nov 03 19:56:36.247422 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.246550 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:36.247422 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.247135 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:36.265272 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.265202 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-87bj5" podStartSLOduration=3.053247744 podStartE2EDuration="8.26518253s" totalImagesPullingTime="5.211934786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:36.040784437 +0000 UTC m=+8.627312928" watchObservedRunningTime="2026-11-03 19:56:36.26518253 +0000 UTC m=+8.851711022" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.897573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:36.897700 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.898093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:36.898203 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:36.898260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:36.898370 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:36.898332 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:37.032538 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:37.032016 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" event={"ID":"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8","Type":"ContainerStarted","Data":"e0f9e9ce2529dbb4999cc6e8ade75fb5afa436216d58f4d78ee371c8414b7d8c"} Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:37.542748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:37.542813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:37.542851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.542975 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543035 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:45.543017185 +0000 UTC m=+18.129545670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543247 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543311 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:45.543297121 +0000 UTC m=+18.129825605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:37.543380 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543389 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:37.543902 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543408 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:37.543902 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543421 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:37.543902 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:37.543461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:45.543449443 +0000 UTC m=+18.129977925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:38.048837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.048374 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:38.050955 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.050789 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:56:38.817114 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.816997 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:56:38.050800873Z","UUID":"0396c634-25f1-455a-81ab-dc7c9a86d503","Handler":null,"Name":"","Endpoint":""} Nov 03 19:56:38.821444 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.821233 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:56:38.821561 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.821459 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:56:38.897405 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.897380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:38.897537 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.897476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:38.897606 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:38.897588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:38.897655 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:38.897601 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:38.897731 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:38.897712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:38.897821 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:38.897804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:39.052275 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.052238 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v2tvg" event={"ID":"3371a194-9a9e-4a3a-9af3-48d86152b1dc","Type":"ContainerStarted","Data":"7919497d7b9256d8fcf0a35ab03737f18d185672637baf5cd13cc666f209c6ab"} Nov 03 19:56:39.054664 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.054530 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="f847ccae830132cf852c8079bb8885ca4a5c1604a8c71c5dd9357b8ce889e48f" exitCode=0 Nov 03 19:56:39.054664 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.054613 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"f847ccae830132cf852c8079bb8885ca4a5c1604a8c71c5dd9357b8ce889e48f"} Nov 03 19:56:39.056563 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.056541 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndfvw" event={"ID":"32c657c7-cf56-4ca8-b035-fd7c55fc2480","Type":"ContainerStarted","Data":"a63663cf1d3c61dbd31ad8ab203d164774b76c019ad73b09420876f5a8fc1534"} Nov 03 19:56:39.059434 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.059407 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" event={"ID":"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8","Type":"ContainerStarted","Data":"d4adff94e1ac860f46affe6c710302cfa4910874ef6c0e24ef752b35b28d904a"} Nov 03 19:56:39.069668 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.069506 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v2tvg" podStartSLOduration=2.549057166 podStartE2EDuration="11.069489111s" totalImagesPullingTime="8.520431945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:39.068750079 +0000 UTC m=+11.655278574" watchObservedRunningTime="2026-11-03 19:56:39.069489111 +0000 UTC m=+11.656017603" Nov 03 19:56:39.111673 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:39.111592 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ndfvw" podStartSLOduration=2.9186316310000002 podStartE2EDuration="11.111578886s" totalImagesPullingTime="8.192947255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:39.11152675 +0000 UTC m=+11.698055243" watchObservedRunningTime="2026-11-03 19:56:39.111578886 +0000 UTC m=+11.698107378" Nov 03 19:56:40.065195 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.065093 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="a9cc1ed23dda5efdb4235e6686ee07308c9fb8bdc368a238adc0a5b4beeb3f59" exitCode=0 Nov 03 19:56:40.065752 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.065206 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"a9cc1ed23dda5efdb4235e6686ee07308c9fb8bdc368a238adc0a5b4beeb3f59"} Nov 03 19:56:40.067840 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.067812 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" event={"ID":"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8","Type":"ContainerStarted","Data":"06745c40fd82d25e92efaba5789aabb3497a127f08576d33bdd9ba2b2bda35a4"} Nov 03 19:56:40.069615 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.069590 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnxjb" event={"ID":"945ef139-c1e8-4154-b869-f18adcc62d8a","Type":"ContainerStarted","Data":"1b65f5d7fd9e8f586b9f916d028b024e74d1428db381f1b5c4cbb46ac7204b4d"} Nov 03 19:56:40.107634 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.107582 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tnxjb" podStartSLOduration=3.675193431 podStartE2EDuration="12.107564976s" totalImagesPullingTime="8.432371545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:40.106667104 +0000 UTC m=+12.693195594" watchObservedRunningTime="2026-11-03 19:56:40.107564976 +0000 UTC m=+12.694093467" Nov 03 19:56:40.126911 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.126863 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" podStartSLOduration=3.561988608 podStartE2EDuration="12.126850617s" totalImagesPullingTime="8.564862009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:40.1267345 +0000 UTC m=+12.713262990" watchObservedRunningTime="2026-11-03 19:56:40.126850617 +0000 UTC m=+12.713379135" Nov 03 19:56:40.897757 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.897726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:40.897757 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.897751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:40.897982 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:40.897897 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:40.897982 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:40.897919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:40.898085 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:40.898044 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:40.898179 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:40.898160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:41.075099 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:41.074798 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerStarted","Data":"23483a5bb979ba1884c61ad0bc25eae681091b5f8892697622ec62f8c8c07b43"} Nov 03 19:56:41.076332 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:41.076304 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-49krk" event={"ID":"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9","Type":"ContainerStarted","Data":"f2b7b98c1ad89712b73495bc682f2276672b309ce06fd99224d0ea87da2ffe9a"} Nov 03 19:56:41.120573 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:41.120518 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-49krk" podStartSLOduration=2.504371459 podStartE2EDuration="13.120499548s" totalImagesPullingTime="10.616128089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:41.120203338 +0000 UTC m=+13.706731828" watchObservedRunningTime="2026-11-03 19:56:41.120499548 +0000 UTC m=+13.707028039" Nov 03 19:56:42.081341 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:42.081297 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="23483a5bb979ba1884c61ad0bc25eae681091b5f8892697622ec62f8c8c07b43" exitCode=0 Nov 03 19:56:42.081957 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:42.081397 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"23483a5bb979ba1884c61ad0bc25eae681091b5f8892697622ec62f8c8c07b43"} Nov 03 19:56:42.898609 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:42.898370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:42.898765 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:42.898446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:42.898765 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:42.898719 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:42.898874 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:42.898804 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:42.898874 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:42.898457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:42.898969 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:42.898878 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:44.897937 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:44.897901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:44.897937 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:44.897920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:44.898573 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:44.897901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:44.898573 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:44.898007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:44.898573 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:44.898095 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:44.898573 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:44.898188 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:45.090662 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.090631 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="458d953385345ea007c281a3d488d0f5f1231a7bd1b28204dd5e0b0e38ec02b4" exitCode=0 Nov 03 19:56:45.090807 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.090683 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"458d953385345ea007c281a3d488d0f5f1231a7bd1b28204dd5e0b0e38ec02b4"} Nov 03 19:56:45.607645 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.607621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:45.607737 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.607660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:45.607737 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.607689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607749 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607784 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607798 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607805 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.607790883 +0000 UTC m=+34.194319363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607810 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:45.607831 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607818 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:45.608033 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607850 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.607838518 +0000 UTC m=+34.194366999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:45.608033 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:45.607874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:01.607857661 +0000 UTC m=+34.194386132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:45.684931 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.684758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:45.839671 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:45.839647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:46.095448 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.095366 2579 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="61940c346c408871137491f77e656130837b059388ac414303b8385608edcc1f" exitCode=0 Nov 03 19:56:46.095970 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.095448 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerDied","Data":"61940c346c408871137491f77e656130837b059388ac414303b8385608edcc1f"} Nov 03 19:56:46.098498 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098480 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"c4e2086e757cc8996f12c341002e0a5b899a6f9306cc4611656e44f98b4a54ac"} Nov 03 19:56:46.098549 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098536 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"c4cc839b73fc6a7d6c2a4e7d298490c135a59518cb29e5b22168a0a412146319"} Nov 03 19:56:46.098587 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098565 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"2ebfa2539a6fdff378fbb900f742ee56d4fd1d2c3c689955c995fec2e290b5bf"} Nov 03 19:56:46.098617 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098594 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"40921b45b98b526013bbf8f0718aba255bb85d23f117fc75a34f0d7470dc7c89"} Nov 03 19:56:46.098656 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098620 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"ad5415c867a54ef5eeb4b76d391fd019e1e7209600016ea45807697b341a263f"} Nov 03 19:56:46.098656 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.098648 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"4e1792c48edd631fbf5846231acbceb0cd344c1246af7f88a6338311fc81c84a"} Nov 03 19:56:46.897933 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.897887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:46.897933 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.897888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:46.898166 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:46.897997 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:46.898166 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:46.898008 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:46.898166 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:46.898073 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:46.898257 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:46.898163 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:47.103700 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:47.103665 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njwcg" event={"ID":"9edf52cd-afe4-46cb-b9b3-148d5445ca6e","Type":"ContainerStarted","Data":"ba014a694629225a8413b20a3d1fa9397f29b80ce7fd42a3d9ee2730e2b4ab94"} Nov 03 19:56:47.131320 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:47.131266 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-njwcg" podStartSLOduration=8.136377008 podStartE2EDuration="19.131252831s" totalImagesPullingTime="10.994875823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:47.130850494 +0000 UTC m=+19.717378995" watchObservedRunningTime="2026-11-03 19:56:47.131252831 +0000 UTC m=+19.717781360" Nov 03 19:56:47.413176 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:47.413144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:47.514583 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:47.514550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:48.109270 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:48.109236 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"103ef55155ff3837280662b371ebc5117625824d62a4f51ea66ab501a31a58bf"} Nov 03 19:56:48.897722 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:48.897680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:48.897922 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:48.897793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:48.897922 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:48.897801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:48.897922 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:48.897825 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:48.897922 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:48.897909 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:48.898100 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:48.897974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:49.242252 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:49.242172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:49.341968 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:49.341939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:50.118174 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.117976 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" event={"ID":"06728ec3-24de-459c-b966-8b88db3bc248","Type":"ContainerStarted","Data":"6b99fff609ed27eb2381c89cb02986919cc934f91d2eebe33f218687f0a6aad5"} Nov 03 19:56:50.118354 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.118228 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:50.118354 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.118279 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:50.135250 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.135221 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:50.151487 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.151439 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" podStartSLOduration=6.746030427 podStartE2EDuration="22.151426394s" totalImagesPullingTime="15.405395967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:50.151406894 +0000 UTC m=+22.737935383" watchObservedRunningTime="2026-11-03 19:56:50.151426394 +0000 UTC m=+22.737954881" Nov 03 19:56:50.569943 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.569911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:50.667637 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.667603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:50.897477 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.897382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:50.897477 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.897419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:50.897670 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:50.897486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:50.897670 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:50.897525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:50.897670 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:50.897599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:50.897769 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:50.897694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:51.121131 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.121085 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:51.135642 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.135618 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:56:51.166039 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.165961 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:51.166183 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.166101 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:51.166659 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.166641 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:56:51.212190 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.212156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:51.314671 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.314516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:51.750741 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.750667 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lb594"] Nov 03 19:56:51.751171 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.750813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:51.751171 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:51.750913 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:51.754635 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.754602 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r97c6"] Nov 03 19:56:51.754774 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.754726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:51.754837 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:51.754818 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:51.755430 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.755404 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-psmsw"] Nov 03 19:56:51.755522 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:51.755511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:51.755615 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:51.755597 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:52.045218 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.045140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:52.139894 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.139860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:52.391483 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.391401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:52.490863 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.490830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:52.590967 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.590935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:52.898315 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:52.898292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:52.898693 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:52.898396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:53.101232 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.101201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:53.205149 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.205053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:53.451840 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.451795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:53.552401 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.552280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:53.769788 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.769756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:53.898022 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.897945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:53.898178 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:53.898062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:53.898178 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:53.898150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:53.898307 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:53.898241 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:54.897857 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:54.897785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:54.898239 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:54.897881 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:54.908687 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:54.908662 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2tvg_3371a194-9a9e-4a3a-9af3-48d86152b1dc/dns-node-resolver/0.log" Nov 03 19:56:55.508681 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:55.508649 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-87bj5_c60f9fad-b8c0-4d6e-b755-c2bda3f2d014/node-ca/0.log" Nov 03 19:56:55.897653 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:55.897576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:55.897787 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:55.897709 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:55.897787 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:55.897763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:55.897866 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:55.897854 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:56.071761 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:56.071721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:56.175980 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:56.175886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:56.278220 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:56.278191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:56.897959 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:56.897933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:56.898146 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:56.898024 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:57.899372 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:57.899142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:57.899753 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:57.899189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:57.899753 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:57.899462 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:57.899753 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:57.899532 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:58.898418 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:58.898383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:56:58.898584 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:58.898482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:56:59.391231 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.391178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:59.490663 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.490626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:59.591576 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.591531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:59.849643 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.849568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:56:59.897641 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.897616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:56:59.897741 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:59.897723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:56:59.897788 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.897772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:56:59.897879 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:56:59.897864 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:56:59.964437 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:56:59.964404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:00.446073 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:00.446040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:00.847450 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:00.847334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:00.897771 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:00.897748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:00.897880 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:00.897845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:01.044176 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.044144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:01.244352 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.244315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:01.618582 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.618557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.618600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.618629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618715 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618722 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618734 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618750 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618760 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4gfwm for pod openshift-network-diagnostics/network-check-target-r97c6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618764 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs podName:471a887a-a065-4aa3-9063-555b93a9a258 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:33.618751174 +0000 UTC m=+66.205279642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs") pod "network-metrics-daemon-psmsw" (UID: "471a887a-a065-4aa3-9063-555b93a9a258") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618778 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret podName:18e97b6a-be41-42af-90ec-bd56e6d791d6 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:33.61877078 +0000 UTC m=+66.205299248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret") pod "global-pull-secret-syncer-lb594" (UID: "18e97b6a-be41-42af-90ec-bd56e6d791d6") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:57:01.618986 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.618796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm podName:47f13c13-8d81-472c-8a39-2f14c8da0242 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:33.618785042 +0000 UTC m=+66.205313510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4gfwm" (UniqueName: "kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm") pod "network-check-target-r97c6" (UID: "47f13c13-8d81-472c-8a39-2f14c8da0242") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:57:01.648752 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.648715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:01.897478 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.897395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:01.897608 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:01.897395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:01.897608 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.897537 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:01.897608 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:01.897574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:02.452073 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.452037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:02.558134 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.558092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:02.703506 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.703427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:02.845776 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.845747 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:02.897487 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.897448 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:02.897662 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:02.897565 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:02.941435 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:02.941408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:03.043329 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:03.043238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:03.247949 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:03.247916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:03.647931 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:03.647884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:03.897824 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:03.897798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:03.898259 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:03.897896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:03.898259 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:03.897951 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:03.898259 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:03.898041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:04.444127 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:04.444075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:04.645397 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:04.645352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:04.862824 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:04.862798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:04.898243 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:04.898216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:04.898572 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:04.898316 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:05.045190 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.045156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:05.246324 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.246202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:05.841960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.841912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:05.898263 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.898229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:05.898650 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:05.898341 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:05.898650 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.898387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:05.898650 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:05.898478 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:05.995788 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:05.995754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:06.444320 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:06.444248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:06.644743 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:06.644709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:06.897551 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:06.897519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:06.897709 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:06.897643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:07.043100 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:07.043050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:07.245201 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:07.245075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:07.847568 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:07.847522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:07.898987 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:07.898948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:07.899227 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:07.899126 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:07.899227 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:07.899148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:07.899341 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:07.899249 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:08.448030 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:08.447989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:08.649855 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:08.649812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:08.898103 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:08.898071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:08.898258 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:08.898198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:09.045807 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:09.045775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:09.245232 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:09.245128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:09.845865 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:09.845820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:09.898328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:09.898299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:09.898447 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:09.898336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:09.898447 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:09.898425 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:09.898558 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:09.898539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:10.041833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:10.041790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:10.244986 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:10.244931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:10.898381 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:10.898346 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:10.898808 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:10.898460 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:11.065969 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.065930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:11.164916 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.164828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:11.738488 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.738454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:11.841811 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.841777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:11.898280 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.898248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:11.898458 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.898323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:11.898458 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:11.898438 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:11.898845 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:11.898576 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:11.946007 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:11.945979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.051556 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.051472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.245797 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.245754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.455640 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.455608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.551978 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.551946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.672751 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.672722 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.773210 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.773129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:12.898223 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:12.898186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:12.898396 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:12.898319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:13.561073 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:13.561037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:13.675030 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:13.674988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:13.897546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:13.897510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:13.897732 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:13.897560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:13.897732 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:13.897655 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-psmsw" podUID="471a887a-a065-4aa3-9063-555b93a9a258" Nov 03 19:57:13.897845 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:13.897748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lb594" podUID="18e97b6a-be41-42af-90ec-bd56e6d791d6" Nov 03 19:57:13.947801 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:13.947738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:14.047627 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:14.047600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:14.898419 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:14.898382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:14.898859 ip-10-0-135-70 kubenswrapper[2579]: E1103 19:57:14.898510 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r97c6" podUID="47f13c13-8d81-472c-8a39-2f14c8da0242" Nov 03 19:57:15.076895 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.076757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:15.165552 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.165523 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeReady" Nov 03 19:57:15.165708 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.165649 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:57:15.176302 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.176273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:15.215137 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.215086 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-chgrg"] Nov 03 19:57:15.217025 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.217005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.217933 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.217909 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8slb6"] Nov 03 19:57:15.219427 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.219409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.222528 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.222500 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:57:15.222625 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.222567 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:57:15.222625 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.222623 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:57:15.222695 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.222642 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:57:15.222748 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.222737 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:57:15.223451 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.223435 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:57:15.223550 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.223484 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:57:15.228511 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.228487 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chgrg"] Nov 03 19:57:15.231476 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.230991 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8slb6"] Nov 03 19:57:15.238774 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.238745 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ltbwq"] Nov 03 19:57:15.240661 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.240635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.256546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.256517 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltbwq"] Nov 03 19:57:15.318096 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.318268 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.318329 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlsx\" (UniqueName: \"kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.318329 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.318329 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.318417 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.318397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsbl\" (UniqueName: \"kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.323000 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.322965 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:57:15.323000 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.322986 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:57:15.323000 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.322969 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:57:15.323188 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.322971 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:57:15.323188 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.323029 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:57:15.418991 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.418943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.419203 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsbl\" (UniqueName: \"kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.419203 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.419203 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.419203 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.419402 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg24s\" (UniqueName: \"kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.419402 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.419402 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.419546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlsx\" (UniqueName: \"kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.419546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.419546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.419546 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419511 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.419980 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.419958 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.423167 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.423145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.423260 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.423194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.431582 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.431560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsbl\" (UniqueName: \"kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.431705 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.431614 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlsx\" (UniqueName: \"kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.520312 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520498 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520498 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520498 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520498 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg24s\" (UniqueName: \"kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520697 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520633 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520697 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.520998 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.520969 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.522660 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.522642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.532783 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.532761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg24s\" (UniqueName: \"kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.532879 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.532862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:15.535117 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.535088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:15.551933 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.551906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:15.821975 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.821893 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8slb6"] Nov 03 19:57:15.824604 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.824441 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chgrg"] Nov 03 19:57:15.825960 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:15.825934 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f8e8c2_cae1_4e2a_92e0_427d4c4f4186.slice/crio-7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695 WatchSource:0}: Error finding container 7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695: Status 404 returned error can't find the container with id 7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695 Nov 03 19:57:15.829920 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:15.829897 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400dd6c5_8a0e_4d2b_a42e_525ba1b4d0b6.slice/crio-4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599 WatchSource:0}: Error finding container 4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599: Status 404 returned error can't find the container with id 4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599 Nov 03 19:57:15.846987 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.846959 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltbwq"] Nov 03 19:57:15.850471 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:15.850440 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2790512d_386a_41e8_8bb6_31ec8faafbc6.slice/crio-e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e WatchSource:0}: Error finding container e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e: Status 404 returned error can't find the container with id e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e Nov 03 19:57:15.897960 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.897936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:15.898139 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.898128 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:15.903135 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.902610 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:57:15.903135 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.902668 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:57:15.903135 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.902668 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:57:15.903135 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:15.902709 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:57:16.175575 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.175541 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltbwq" event={"ID":"2790512d-386a-41e8-8bb6-31ec8faafbc6","Type":"ContainerStarted","Data":"ed1612bf9decf87048f491b8e191331934d39d182b88b8ffb97222a5e2cb8849"} Nov 03 19:57:16.175775 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.175596 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltbwq" event={"ID":"2790512d-386a-41e8-8bb6-31ec8faafbc6","Type":"ContainerStarted","Data":"e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e"} Nov 03 19:57:16.176743 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.176708 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chgrg" event={"ID":"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6","Type":"ContainerStarted","Data":"4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599"} Nov 03 19:57:16.177783 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.177760 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8slb6" event={"ID":"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186","Type":"ContainerStarted","Data":"7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695"} Nov 03 19:57:16.526748 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.526673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:16.628364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.628323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:16.738368 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.738332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:16.759121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.759084 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs"] Nov 03 19:57:16.761933 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.761912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.767738 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.767710 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 19:57:16.767857 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.767799 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:57:16.767857 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.767843 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:57:16.767959 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.767888 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:57:16.767959 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.767952 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 19:57:16.768059 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.768051 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fxkp4\"" Nov 03 19:57:16.773386 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.773202 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs"] Nov 03 19:57:16.837328 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.837288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.837503 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.837337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.837545 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.837496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.837545 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.837531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blbkp\" (UniqueName: \"kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.849137 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.848017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:16.860488 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.860453 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:57:16.862722 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.862697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.867837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.867658 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 19:57:16.867837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.867675 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:57:16.867837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.867660 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 19:57:16.867837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.867718 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 19:57:16.867837 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.867808 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:57:16.868373 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.868352 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 19:57:16.868438 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.868412 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 19:57:16.868538 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.868525 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mcwl7\"" Nov 03 19:57:16.874168 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.874140 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:57:16.898258 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.898222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:16.938273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbkp\" (UniqueName: \"kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.938806 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.938652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:16.939160 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.939139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.948698 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.948663 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.948842 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.948700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.951450 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.951421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:16.953430 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.953404 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbkp\" (UniqueName: \"kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:16.968038 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.967816 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:57:16.968038 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.967832 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:57:16.968038 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:16.968003 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:57:17.039508 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039508 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.039718 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.039653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.040823 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.040305 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.040823 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.040449 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.040823 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.040461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.040823 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.040696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.043086 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.043060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.043501 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.043480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.051589 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.051550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:17.053339 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.053313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.078013 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.077971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:17.178240 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.178210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:17.183233 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.183168 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltbwq" event={"ID":"2790512d-386a-41e8-8bb6-31ec8faafbc6","Type":"ContainerStarted","Data":"fa590298ab8ebb9a208fef0e47c1c4a322beaa18df3aefed9329eb14d43bdc83"} Nov 03 19:57:17.363682 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.363635 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs"] Nov 03 19:57:17.367819 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:17.367787 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc34dba5_14d2_402c_99ed_0a27e607616d.slice/crio-0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97 WatchSource:0}: Error finding container 0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97: Status 404 returned error can't find the container with id 0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97 Nov 03 19:57:17.462734 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:17.462699 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:57:17.468481 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:17.468441 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea0e226_87c0_40c9_b1b3_6029f76e6576.slice/crio-57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c WatchSource:0}: Error finding container 57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c: Status 404 returned error can't find the container with id 57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c Nov 03 19:57:18.189048 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.188917 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b85776785-crvbv" event={"ID":"5ea0e226-87c0-40c9-b1b3-6029f76e6576","Type":"ContainerStarted","Data":"57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c"} Nov 03 19:57:18.192227 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.192197 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chgrg" event={"ID":"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6","Type":"ContainerStarted","Data":"fae0e439c544db202f326f52b8e06e94c99a04498021c2e130eab1dc0232953b"} Nov 03 19:57:18.192346 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.192252 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chgrg" event={"ID":"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6","Type":"ContainerStarted","Data":"ce2841d38d9beb8939a0c4d2305367b8e4159bbfe48e2e7e26896018f671384f"} Nov 03 19:57:18.192979 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.192960 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:18.194658 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.194631 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" event={"ID":"fc34dba5-14d2-402c-99ed-0a27e607616d","Type":"ContainerStarted","Data":"0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97"} Nov 03 19:57:18.214533 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:18.214472 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-chgrg" podStartSLOduration=1.315351661 podStartE2EDuration="3.214452912s" totalImagesPullingTime="1.899101251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:18.212730943 +0000 UTC m=+50.799259435" watchObservedRunningTime="2026-11-03 19:57:18.214452912 +0000 UTC m=+50.800981408" Nov 03 19:57:19.202790 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.202043 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" event={"ID":"fc34dba5-14d2-402c-99ed-0a27e607616d","Type":"ContainerStarted","Data":"b0bbfeb27d9880e93865effbb2624b0cfb45ec9a0dc28a9f31b982e1db13b223"} Nov 03 19:57:19.202790 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.202118 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" event={"ID":"fc34dba5-14d2-402c-99ed-0a27e607616d","Type":"ContainerStarted","Data":"7bd86d4efd775ee746e8c62afe5ac51080f63cb12d3153ea443938088db87c56"} Nov 03 19:57:19.205004 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.204965 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8slb6" event={"ID":"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186","Type":"ContainerStarted","Data":"402ca1474e46c254240bd51e4081137263e03904684e0faae4c1ef8fb16aac1a"} Nov 03 19:57:19.216302 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.214771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltbwq" event={"ID":"2790512d-386a-41e8-8bb6-31ec8faafbc6","Type":"ContainerStarted","Data":"a0e2fbf114fc1e41b5ba9ee8cae50e85a4bb8b463d24de8fbd19cfa9969d6c8c"} Nov 03 19:57:19.225941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.225890 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" podStartSLOduration=1.661305315 podStartE2EDuration="3.225876655s" totalImagesPullingTime="1.56457134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:19.225256313 +0000 UTC m=+51.811784835" watchObservedRunningTime="2026-11-03 19:57:19.225876655 +0000 UTC m=+51.812405144" Nov 03 19:57:19.267375 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.267310 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ltbwq" podStartSLOduration=1.734364402 podStartE2EDuration="4.267291766s" totalImagesPullingTime="2.532927364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:57:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:19.250191538 +0000 UTC m=+51.836720029" watchObservedRunningTime="2026-11-03 19:57:19.267291766 +0000 UTC m=+51.853820250" Nov 03 19:57:19.268525 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.267980 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8slb6" podStartSLOduration=1.120567109 podStartE2EDuration="4.2679709s" totalImagesPullingTime="3.147403791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:19.266777427 +0000 UTC m=+51.853305921" watchObservedRunningTime="2026-11-03 19:57:19.2679709 +0000 UTC m=+51.854499391" Nov 03 19:57:19.292948 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.292909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:19.452916 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.452828 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:57:19.455059 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.455034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.463725 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.463704 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 19:57:19.468531 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.468502 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:57:19.488865 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.488830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:19.562449 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562641 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562641 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562641 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562766 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562766 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562852 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.562852 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.562785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.593366 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.593331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:19.664209 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664209 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664427 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664466 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664517 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664611 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664611 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664739 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.664811 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.664789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.665034 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.665006 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.665164 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.665078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.665472 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.665454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.665589 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.665564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.667274 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.667253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.667388 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.667322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.677175 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.677142 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:19.768385 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:19.768084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:20.047822 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:20.047739 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:57:20.052255 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:20.052227 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488637a9_aaf1_4506_a7ad_5d20665bb414.slice/crio-3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9 WatchSource:0}: Error finding container 3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9: Status 404 returned error can't find the container with id 3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9 Nov 03 19:57:20.219188 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:20.219148 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f46b5684f-zmphc" event={"ID":"488637a9-aaf1-4506-a7ad-5d20665bb414","Type":"ContainerStarted","Data":"3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9"} Nov 03 19:57:20.913370 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:20.913336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.076261 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.076219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.132890 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.132853 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w"] Nov 03 19:57:21.135377 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.135347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.147088 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.147049 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw"] Nov 03 19:57:21.148919 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.148892 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 19:57:21.149036 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.148936 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:57:21.149036 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.149012 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z5cft\"" Nov 03 19:57:21.149557 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.149529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.179375 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.179281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z6gt5\"" Nov 03 19:57:21.179457 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.179402 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:57:21.180144 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.179981 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 19:57:21.181847 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.181822 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w"] Nov 03 19:57:21.183195 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.183170 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 19:57:21.185480 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.185001 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k7sbn"] Nov 03 19:57:21.187391 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.187371 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw"] Nov 03 19:57:21.187533 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.187511 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.190685 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.190650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.226121 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.226071 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b85776785-crvbv" event={"ID":"5ea0e226-87c0-40c9-b1b3-6029f76e6576","Type":"ContainerStarted","Data":"965d861c14fd1ff5092928f4bb9f86c5b9228c0adab48b05844ab3c531733990"} Nov 03 19:57:21.245697 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.245660 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:57:21.245964 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.245942 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fksvw\"" Nov 03 19:57:21.246077 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.245978 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:57:21.248396 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.247343 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b85776785-crvbv" podStartSLOduration=2.174046552 podStartE2EDuration="5.247329041s" totalImagesPullingTime="3.073282489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:21.247194961 +0000 UTC m=+53.833723454" watchObservedRunningTime="2026-11-03 19:57:21.247329041 +0000 UTC m=+53.833857531" Nov 03 19:57:21.256129 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.256092 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:57:21.279049 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.279244 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279244 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279244 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkqf\" (UniqueName: \"kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.279364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.279364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.279600 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.279371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77n8\" (UniqueName: \"kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.290739 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.290703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.380146 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.380348 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380348 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380348 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380348 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380541 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkqf\" (UniqueName: \"kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x8q\" (UniqueName: \"kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77n8\" (UniqueName: \"kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.380831 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.380805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.384016 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.381365 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.384016 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.381983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.384016 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.383411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.384016 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.383902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.384326 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.384098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.384405 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.384381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.384476 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.384381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.384568 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.384551 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.392055 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.391967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.394553 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.394525 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77n8\" (UniqueName: \"kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.394697 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.394641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkqf\" (UniqueName: \"kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.451780 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.451699 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:21.481395 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481395 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481590 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481590 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x8q\" (UniqueName: \"kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481590 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481590 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481590 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481768 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481768 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.481768 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.481701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.486495 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.482572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.486495 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.482718 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.486495 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.483053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.486495 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.483475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.486495 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.483696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.492410 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.491667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:21.492410 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.491994 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.492586 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.492568 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.499553 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.499522 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x8q\" (UniqueName: \"kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.502647 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.502443 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:21.517171 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.516255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.625356 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.625315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.736922 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.736840 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w"] Nov 03 19:57:21.740635 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:21.740606 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3cd7f2_0667_4487_965c_44e36b07ceb3.slice/crio-e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749 WatchSource:0}: Error finding container e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749: Status 404 returned error can't find the container with id e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749 Nov 03 19:57:21.786305 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.786269 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw"] Nov 03 19:57:21.789626 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:21.789600 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e1076f_26b7_40eb_a11f_b9eca41ee6fc.slice/crio-efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef WatchSource:0}: Error finding container efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef: Status 404 returned error can't find the container with id efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef Nov 03 19:57:21.880154 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.880094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:21.983072 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:21.983029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:22.136019 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.135966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:22.217631 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.217598 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:57:22.219968 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.219952 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.228695 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228669 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228736 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228764 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228812 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228855 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.228965 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wbnrm\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.229036 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 19:57:22.229058 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.229063 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 19:57:22.229436 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.229102 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 19:57:22.229436 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.229209 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 19:57:22.234460 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.234432 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7sbn" event={"ID":"426ca1a6-344a-4f33-9e58-50be798ed0a9","Type":"ContainerStarted","Data":"afc10babb45756f066b43615728140b72b64ee217c3b75f59b2c86f5ad4befd4"} Nov 03 19:57:22.238559 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.238421 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" event={"ID":"fc3cd7f2-0667-4487-965c-44e36b07ceb3","Type":"ContainerStarted","Data":"23d8976d3744e1827ce18883a8762a529f428f9390b7a7b1ca3baaa8d5b37f30"} Nov 03 19:57:22.238559 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.238489 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" event={"ID":"fc3cd7f2-0667-4487-965c-44e36b07ceb3","Type":"ContainerStarted","Data":"2ab16a2a573eaf546d8d22bd94fd84b26af94d7e72696fc15d604844f96ae085"} Nov 03 19:57:22.238559 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.238517 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" event={"ID":"fc3cd7f2-0667-4487-965c-44e36b07ceb3","Type":"ContainerStarted","Data":"e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749"} Nov 03 19:57:22.238853 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.238833 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:57:22.241663 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.241612 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f46b5684f-zmphc" event={"ID":"488637a9-aaf1-4506-a7ad-5d20665bb414","Type":"ContainerStarted","Data":"ac1b1f500f9632f9d428cb2ea476327003e4d40ff0bd3ab57d1f54e4156d398e"} Nov 03 19:57:22.244154 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.244097 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" event={"ID":"31e1076f-26b7-40eb-a11f-b9eca41ee6fc","Type":"ContainerStarted","Data":"efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef"} Nov 03 19:57:22.251962 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.251930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:22.294970 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.294903 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f46b5684f-zmphc" podStartSLOduration=1.545702718 podStartE2EDuration="3.294886321s" totalImagesPullingTime="1.749183603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:22.294777809 +0000 UTC m=+54.881306300" watchObservedRunningTime="2026-11-03 19:57:22.294886321 +0000 UTC m=+54.881414812" Nov 03 19:57:22.351684 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.351649 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:22.388507 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.388413 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.388738 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.388714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.388817 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.388767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.388876 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.388820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389024 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.388982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389182 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389182 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389294 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389294 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389384 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389384 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389477 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.389477 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.389443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.489941 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.489897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.489958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.489991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490155 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.490833 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.491095 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.491095 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.490897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.491215 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.491193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.492193 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.491835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.492719 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.492673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.494477 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.494452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.495840 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.495777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.495909 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.495847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.496483 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.496444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.498429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.498364 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.498429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.498373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.498429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.498394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.498429 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.498418 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.504722 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.504700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.505082 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.505063 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.542278 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.542234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:22.850984 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:22.850929 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:57:22.856868 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:22.856831 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adf7348_9f76_4a5c_889a_4f687fe056b2.slice/crio-559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae WatchSource:0}: Error finding container 559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae: Status 404 returned error can't find the container with id 559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae Nov 03 19:57:23.144199 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.144172 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:23.248931 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.248902 2579 generic.go:356] "Generic (PLEG): container finished" podID="426ca1a6-344a-4f33-9e58-50be798ed0a9" containerID="7cc4d081bbb479ff20f803c5ea60e02e195a163400f294d8c5386fb5cf4a9721" exitCode=0 Nov 03 19:57:23.249298 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.249008 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7sbn" event={"ID":"426ca1a6-344a-4f33-9e58-50be798ed0a9","Type":"ContainerDied","Data":"7cc4d081bbb479ff20f803c5ea60e02e195a163400f294d8c5386fb5cf4a9721"} Nov 03 19:57:23.251377 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.251344 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" event={"ID":"fc3cd7f2-0667-4487-965c-44e36b07ceb3","Type":"ContainerStarted","Data":"89c26fe6892f2ff35a70cb849112049c7478cc0f870db1b46e19e34865b138b0"} Nov 03 19:57:23.253166 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.252590 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae"} Nov 03 19:57:23.306838 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:23.306783 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" podStartSLOduration=1.0980440360000001 podStartE2EDuration="2.306762369s" totalImagesPullingTime="1.208718333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:23.306157726 +0000 UTC m=+55.892686216" watchObservedRunningTime="2026-11-03 19:57:23.306762369 +0000 UTC m=+55.893290863" Nov 03 19:57:24.113759 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.113664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:24.219912 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.219869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:24.258264 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.258231 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" event={"ID":"31e1076f-26b7-40eb-a11f-b9eca41ee6fc","Type":"ContainerStarted","Data":"fb6f3175dbf4a8c3ac661a37101416c0a1a5dedcd0b6dc225040fef1096d900c"} Nov 03 19:57:24.258725 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.258294 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" event={"ID":"31e1076f-26b7-40eb-a11f-b9eca41ee6fc","Type":"ContainerStarted","Data":"fe376086b370dc8b456e84de4e56851ff4286b224f757f808d00be534ea9516d"} Nov 03 19:57:24.258725 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.258318 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" event={"ID":"31e1076f-26b7-40eb-a11f-b9eca41ee6fc","Type":"ContainerStarted","Data":"564ff06ad99f9d0238fb37d97cf4c33608cdad1235557cfe5bbc0fefeb4ad959"} Nov 03 19:57:24.259853 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.259831 2579 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="dadada5ebba3a4035f3e88d8407fac4175aca7617af99b5be01b2a474e8fb945" exitCode=0 Nov 03 19:57:24.259946 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.259919 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"dadada5ebba3a4035f3e88d8407fac4175aca7617af99b5be01b2a474e8fb945"} Nov 03 19:57:24.262487 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.262464 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7sbn" event={"ID":"426ca1a6-344a-4f33-9e58-50be798ed0a9","Type":"ContainerStarted","Data":"5d356ed12210986c41b159586408a1a522a4e15d851386db1253d7a5a108f13a"} Nov 03 19:57:24.262575 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.262513 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7sbn" event={"ID":"426ca1a6-344a-4f33-9e58-50be798ed0a9","Type":"ContainerStarted","Data":"0f973409c0b981a43fb5a1402862032cedfc51ec599503f1c48d924fbd1a5333"} Nov 03 19:57:24.280289 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.280239 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" podStartSLOduration=1.875451441 podStartE2EDuration="3.280225247s" totalImagesPullingTime="1.404773806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:24.279035004 +0000 UTC m=+56.865563495" watchObservedRunningTime="2026-11-03 19:57:24.280225247 +0000 UTC m=+56.866753734" Nov 03 19:57:24.327714 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.327671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:24.336787 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.336743 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k7sbn" podStartSLOduration=1.890419451 podStartE2EDuration="3.336730812s" totalImagesPullingTime="1.363471498s" totalInitContainerRuntime="82.839863ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:24.335246793 +0000 UTC m=+56.921775284" watchObservedRunningTime="2026-11-03 19:57:24.336730812 +0000 UTC m=+56.923259301" Nov 03 19:57:24.427373 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.427292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:24.678635 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.678548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:24.778957 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:24.778917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:25.531963 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.530460 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5687c6f496-d89k4"] Nov 03 19:57:25.533895 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.533868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.539752 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.539725 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 19:57:25.539752 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.539726 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:57:25.544959 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.544935 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5687c6f496-d89k4"] Nov 03 19:57:25.574193 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.574168 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 19:57:25.574315 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.574235 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 19:57:25.574315 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.574235 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bgwmm\"" Nov 03 19:57:25.574315 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.574281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f3423uefgu8jc\"" Nov 03 19:57:25.623638 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.623745 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.623820 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.623873 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.623873 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.623992 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.623978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zkt\" (UniqueName: \"kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.624051 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.624015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zkt\" (UniqueName: \"kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726206 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.725789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726849 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.726822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.726984 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.726960 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.728778 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.728741 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.729602 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.729558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.735406 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.734593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.738440 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.738176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zkt\" (UniqueName: \"kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.844375 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.844340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:25.941074 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.941043 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:57:25.947585 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.947550 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-52pfh"] Nov 03 19:57:25.949957 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.949930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:25.958460 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.958432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:25.959201 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.959174 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 19:57:25.959201 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.959179 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-llgd5\"" Nov 03 19:57:25.960451 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.960428 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-52pfh"] Nov 03 19:57:25.974998 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.974961 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:57:25.978407 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.977738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:25.988182 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:25.988157 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:57:26.028338 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.028338 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028563 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028563 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028563 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh7nl\" (UniqueName: \"kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.028700 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028700 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028700 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028788 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.028817 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.028773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.055741 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.055644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:26.129984 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.129945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.129984 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.129985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh7nl\" (UniqueName: \"kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130273 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130585 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130791 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.130915 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.130889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.131158 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.131132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.131257 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.131175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.131257 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.131233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.133134 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.133021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.133243 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.133177 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.133374 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.133358 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.134694 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.134588 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5687c6f496-d89k4"] Nov 03 19:57:26.136813 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:26.136790 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dccc20_8871_4155_aa48_7ffbf940209f.slice/crio-72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b WatchSource:0}: Error finding container 72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b: Status 404 returned error can't find the container with id 72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b Nov 03 19:57:26.168980 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.168944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh7nl\" (UniqueName: \"kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.169100 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.168985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.182536 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.182511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:26.264013 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.263965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:26.270433 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.270396 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" event={"ID":"78dccc20-8871-4155-aa48-7ffbf940209f","Type":"ContainerStarted","Data":"72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b"} Nov 03 19:57:26.274073 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.274046 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"3cd18385c076dc354a28253a87a6afad04c3437655dbd2161c8aa2ea5ef07d05"} Nov 03 19:57:26.274240 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.274161 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"b6fa5b8aa2b08aff66e9a3fa686105c0c5c6d6db5dd905248769fdecea990e1e"} Nov 03 19:57:26.274240 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.274208 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"18ab646cae9c0d881559e7ba04f7f42edb2e03d81befcc89a85639d0188a3d24"} Nov 03 19:57:26.274357 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.274246 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"eec2ed1542d04fd24ffddc22553e3dd47bc7d254604dad49989e148a6853e9b8"} Nov 03 19:57:26.274357 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.274286 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"f1caaa4597cd1d72fb8bba5c95b4de86e587a36dd42bdb9400b65c478d62bcd8"} Nov 03 19:57:26.288296 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.288265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:26.552544 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.552509 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-52pfh"] Nov 03 19:57:26.555781 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:26.555756 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f24aac_5672_4b00_b199_324e3da68852.slice/crio-fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a WatchSource:0}: Error finding container fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a: Status 404 returned error can't find the container with id fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a Nov 03 19:57:26.594399 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:26.594380 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:57:26.616759 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:26.616719 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ac0444_b127_4046_80b2_49f8d1a416e5.slice/crio-2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2 WatchSource:0}: Error finding container 2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2: Status 404 returned error can't find the container with id 2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2 Nov 03 19:57:27.178408 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.178368 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:27.284487 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.284435 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerStarted","Data":"35718d8b301a82b15b3650cb427837a2c611b697b68c0ea4d80f82c3bdfd8cde"} Nov 03 19:57:27.286470 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.286434 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" event={"ID":"73f24aac-5672-4b00-b199-324e3da68852","Type":"ContainerStarted","Data":"fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a"} Nov 03 19:57:27.288474 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.288394 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b794d6b84-mrbt9" event={"ID":"a0ac0444-b127-4046-80b2-49f8d1a416e5","Type":"ContainerStarted","Data":"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb"} Nov 03 19:57:27.288474 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.288447 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b794d6b84-mrbt9" event={"ID":"a0ac0444-b127-4046-80b2-49f8d1a416e5","Type":"ContainerStarted","Data":"2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2"} Nov 03 19:57:27.321176 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.321097 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.21072409 podStartE2EDuration="5.321078162s" totalImagesPullingTime="3.097585776s" totalInitContainerRuntime="12.768296ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:27.320874829 +0000 UTC m=+59.907403316" watchObservedRunningTime="2026-11-03 19:57:27.321078162 +0000 UTC m=+59.907606653" Nov 03 19:57:27.345364 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.344726 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b794d6b84-mrbt9" podStartSLOduration=2.344711798 podStartE2EDuration="2.344711798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:27.343605397 +0000 UTC m=+59.930133900" watchObservedRunningTime="2026-11-03 19:57:27.344711798 +0000 UTC m=+59.931240289" Nov 03 19:57:27.436441 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.433584 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:57:27.449267 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.448181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.456788 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.456760 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.456812 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.456912 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457056 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457094 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457144 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457217 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:57:27.457292 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457266 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:57:27.457701 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457481 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-648mvoststcbk\"" Nov 03 19:57:27.457701 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457590 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjsd2\"" Nov 03 19:57:27.457801 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457730 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:57:27.457880 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.457856 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:57:27.483529 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.483284 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:57:27.483529 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.483490 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:57:27.487462 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.487435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:27.548125 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548294 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548294 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548294 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548419 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548419 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548497 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.548535 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548893 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.549097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.548977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.553885 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.553860 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:57:27.554324 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.554055 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:57:27.554324 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.554059 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:57:27.650264 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650424 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650672 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.650905 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.650865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.652232 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.651518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.652232 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.651565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.652232 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.652188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.653583 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.653535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.654496 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.654459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.655250 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.655224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.655352 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.655263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.655819 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.655767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.655819 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.655801 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.656221 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.656182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.656750 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.656707 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.657375 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.657333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.657947 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.657907 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.659550 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.659532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.661593 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.661569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.662024 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.662003 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.662538 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.662494 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.665205 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.664880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.667196 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.667170 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.668243 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.668198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:27.804581 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.804489 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjsd2\"" Nov 03 19:57:27.818693 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:27.818438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:28.135340 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.135297 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:57:28.139983 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:28.139943 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad270548_2169_4822_8c43_17f547dc0421.slice/crio-78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7 WatchSource:0}: Error finding container 78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7: Status 404 returned error can't find the container with id 78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7 Nov 03 19:57:28.293093 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.293064 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" event={"ID":"78dccc20-8871-4155-aa48-7ffbf940209f","Type":"ContainerStarted","Data":"5d891c18ab42028499369bfa62dbb23cd1b3e337f867759598687a784d66c98c"} Nov 03 19:57:28.294675 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.294655 2579 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" exitCode=0 Nov 03 19:57:28.294783 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.294695 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} Nov 03 19:57:28.294783 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.294770 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7"} Nov 03 19:57:28.313640 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.313596 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" podStartSLOduration=1.2685474509999999 podStartE2EDuration="3.313583087s" totalImagesPullingTime="2.045035636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:28.31335886 +0000 UTC m=+60.899887350" watchObservedRunningTime="2026-11-03 19:57:28.313583087 +0000 UTC m=+60.900111577" Nov 03 19:57:28.680743 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.680710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:28.782201 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:28.782159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:29.075177 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.075079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:29.222490 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.222446 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:29.301836 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.301796 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" event={"ID":"73f24aac-5672-4b00-b199-324e3da68852","Type":"ContainerStarted","Data":"f968b354f856f30c76d0edc4df00fc44a251771caffbca17d47520fd39f379c7"} Nov 03 19:57:29.321014 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.320962 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" podStartSLOduration=2.614425449 podStartE2EDuration="4.320944196s" totalImagesPullingTime="1.706518747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:29.319767784 +0000 UTC m=+61.906296275" watchObservedRunningTime="2026-11-03 19:57:29.320944196 +0000 UTC m=+61.907472689" Nov 03 19:57:29.380789 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.380752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:29.480575 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.480540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:29.769018 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.768921 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:29.769018 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.769002 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:29.774253 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:29.774226 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:30.306283 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:30.306252 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:30.311536 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:30.311507 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:30.311690 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:30.311581 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:32.319340 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:32.319301 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} Nov 03 19:57:32.319700 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:32.319375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} Nov 03 19:57:33.715555 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.715513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:33.716067 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.715568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:33.716067 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.715655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:33.718794 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.718765 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:57:33.720013 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.719980 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:57:33.720141 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.720020 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:57:33.728640 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.728610 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:33.728882 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.728853 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:57:33.728977 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.728914 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:33.739959 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.739923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:33.915060 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.915038 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:57:33.919093 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.919073 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:57:33.922179 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.922156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:33.926286 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:33.926118 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:34.015535 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.015366 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:57:34.028042 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.026072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:34.215842 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.215823 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lb594"] Nov 03 19:57:34.218652 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:34.218617 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e97b6a_be41_42af_90ec_bd56e6d791d6.slice/crio-07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739 WatchSource:0}: Error finding container 07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739: Status 404 returned error can't find the container with id 07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739 Nov 03 19:57:34.219097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.219067 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-psmsw"] Nov 03 19:57:34.222002 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:34.221978 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471a887a_a065_4aa3_9063_555b93a9a258.slice/crio-caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0 WatchSource:0}: Error finding container caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0: Status 404 returned error can't find the container with id caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0 Nov 03 19:57:34.315650 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.315548 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r97c6"] Nov 03 19:57:34.319141 ip-10-0-135-70 kubenswrapper[2579]: W1103 19:57:34.319095 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f13c13_8d81_472c_8a39_2f14c8da0242.slice/crio-c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be WatchSource:0}: Error finding container c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be: Status 404 returned error can't find the container with id c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be Nov 03 19:57:34.329695 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.329662 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lb594" event={"ID":"18e97b6a-be41-42af-90ec-bd56e6d791d6","Type":"ContainerStarted","Data":"07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739"} Nov 03 19:57:34.333862 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.333835 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} Nov 03 19:57:34.333970 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.333909 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} Nov 03 19:57:34.333970 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.333953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} Nov 03 19:57:34.334059 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.333992 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerStarted","Data":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} Nov 03 19:57:34.335163 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.335136 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r97c6" event={"ID":"47f13c13-8d81-472c-8a39-2f14c8da0242","Type":"ContainerStarted","Data":"c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be"} Nov 03 19:57:34.336275 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.336256 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-psmsw" event={"ID":"471a887a-a065-4aa3-9063-555b93a9a258","Type":"ContainerStarted","Data":"caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0"} Nov 03 19:57:34.373812 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.373726 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.899637126 podStartE2EDuration="7.373705621s" totalImagesPullingTime="5.454337726s" totalInitContainerRuntime="19.730769ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:34.372235121 +0000 UTC m=+66.958763615" watchObservedRunningTime="2026-11-03 19:57:34.373705621 +0000 UTC m=+66.960234112" Nov 03 19:57:34.554057 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.554013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:34.657908 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:34.657850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:36.030199 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.030099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:36.131365 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.131328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:36.288737 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.288639 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:36.289097 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.289045 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:36.295443 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.295414 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:36.347631 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.347591 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-psmsw" event={"ID":"471a887a-a065-4aa3-9063-555b93a9a258","Type":"ContainerStarted","Data":"3e44084e0599152ae0f3fd91f08a328dc9ca3aec92876bb73f77366448276e46"} Nov 03 19:57:36.347815 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.347657 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-psmsw" event={"ID":"471a887a-a065-4aa3-9063-555b93a9a258","Type":"ContainerStarted","Data":"f602f25299a0419b622b4ce38b3ea23a62bb142c5a9058b134a28c6334163654"} Nov 03 19:57:36.352439 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.352413 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:36.368365 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.368306 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-psmsw" podStartSLOduration=67.120334059 podStartE2EDuration="1m8.368287011s" totalImagesPullingTime="1.247952952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:36.365938177 +0000 UTC m=+68.952466668" watchObservedRunningTime="2026-11-03 19:57:36.368287011 +0000 UTC m=+68.954815501" Nov 03 19:57:36.416656 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:36.416627 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:57:37.819136 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:37.819088 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:38.360216 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:38.360135 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r97c6" event={"ID":"47f13c13-8d81-472c-8a39-2f14c8da0242","Type":"ContainerStarted","Data":"83665f286ae0ffbca15c194fa5df9d0ab207fa86d0e8ddd7edb4e4cf269bbdcd"} Nov 03 19:57:39.207464 ip-10-0-135-70 kubenswrapper[2579]: I1103 19:57:39.207084 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:39.207241 ip-10-0-135-70 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:57:39.223565 ip-10-0-135-70 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:57:39.223734 ip-10-0-135-70 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:57:39.223786 ip-10-0-135-70 systemd[1]: kubelet.service: Consumed 4.940s CPU time. Nov 03 19:57:39.234646 ip-10-0-135-70 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:57:39.304621 ip-10-0-135-70 kubenswrapper[5954]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:39.304621 ip-10-0-135-70 kubenswrapper[5954]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:57:39.304621 ip-10-0-135-70 kubenswrapper[5954]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:39.304621 ip-10-0-135-70 kubenswrapper[5954]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:39.307011 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.306912 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.307011 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307006 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.307011 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307012 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.307011 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307017 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307022 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307027 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307030 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307034 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307038 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307042 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307046 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307051 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307055 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307059 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307064 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307069 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307073 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307077 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307081 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307086 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307090 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307094 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307098 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.307311 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307118 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307124 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307128 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307132 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307136 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307140 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307144 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307149 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307153 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307157 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307162 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307166 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307171 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307175 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307179 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307183 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307188 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307192 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307196 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307201 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.308182 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307205 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307209 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307214 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307218 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307222 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307228 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307233 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307237 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307241 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307246 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307250 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307254 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307259 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307263 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307267 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307272 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307276 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307280 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307285 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.309040 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307289 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307293 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307298 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307302 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307306 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307313 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307317 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307321 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307326 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307331 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307336 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307340 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307344 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307349 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307354 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307360 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307365 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307372 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307379 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.309894 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307384 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307390 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307396 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307400 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307405 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307410 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307414 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307419 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307423 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307428 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307433 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307438 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307443 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307447 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307451 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307456 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307461 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307465 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307471 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.310738 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307475 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307483 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307490 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307495 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307499 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.307504 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308806 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308819 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308825 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308831 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308836 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308841 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308845 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308850 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308855 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308860 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308864 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308868 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308873 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.311591 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308877 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308881 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308886 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308890 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308898 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308904 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308911 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308917 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308922 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308928 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308932 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308937 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308942 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308946 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308951 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308955 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308960 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308965 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.312445 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308969 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308973 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308978 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308982 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308987 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308992 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.308996 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309001 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309006 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309011 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309016 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309020 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309025 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309029 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309034 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309038 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309043 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309047 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309051 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309055 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.313249 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309059 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309064 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309069 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309073 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309078 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309082 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309088 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309093 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309098 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309119 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309124 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309128 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309133 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309138 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309142 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309147 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309151 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309155 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309160 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309164 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.314035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309169 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309174 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309178 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309200 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309207 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309211 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309216 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309221 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309225 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309230 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309234 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309238 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309243 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309247 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309252 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309256 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309260 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309265 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309270 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309274 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.314556 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309279 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309283 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309287 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309295 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309299 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309304 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309308 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309312 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309317 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309321 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309325 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309330 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309336 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309341 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.309345 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310200 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310210 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310214 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310219 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310223 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.315083 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310228 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310233 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310238 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310242 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310247 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310251 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310256 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310260 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310265 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310271 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310278 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310285 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310291 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310295 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310300 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310307 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310311 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310315 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310320 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.315651 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310324 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310329 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310333 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310338 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310342 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310347 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310351 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310355 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310359 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310365 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310369 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310374 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310378 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310382 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310387 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310391 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310395 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310399 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310404 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310409 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.316385 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310413 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310418 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310422 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310427 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310431 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310436 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310441 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310445 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310454 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310459 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310463 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310467 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310472 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310476 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310481 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310485 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310489 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310493 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310498 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310503 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.317181 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310507 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310512 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310525 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310538 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310543 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310547 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310551 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310556 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310561 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310565 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310570 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310574 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310579 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310583 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310587 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310592 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310599 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310606 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310611 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.317692 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310615 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310622 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310626 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310631 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310635 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310639 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310644 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310648 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310653 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310657 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310661 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310665 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310671 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310676 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310681 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310685 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310698 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310703 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310707 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310711 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.318171 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310716 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310720 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.310725 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310864 5954 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310875 5954 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310891 5954 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310899 5954 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310907 5954 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310912 5954 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310919 5954 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310926 5954 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310931 5954 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310936 5954 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310942 5954 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310948 5954 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310953 5954 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310958 5954 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310963 5954 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310968 5954 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310973 5954 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310977 5954 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310987 5954 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310992 5954 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:57:39.318658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.310998 5954 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311005 5954 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311011 5954 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311017 5954 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311022 5954 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311027 5954 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311033 5954 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311038 5954 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311043 5954 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311048 5954 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311053 5954 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311058 5954 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311065 5954 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311071 5954 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311075 5954 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311081 5954 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311086 5954 flags.go:64] FLAG: --enable-server="true" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311090 5954 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311098 5954 flags.go:64] FLAG: --event-burst="100" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311118 5954 flags.go:64] FLAG: --event-qps="50" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311124 5954 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311129 5954 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311135 5954 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311141 5954 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:57:39.319222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311146 5954 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311152 5954 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311157 5954 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311161 5954 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311166 5954 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311171 5954 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311177 5954 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311182 5954 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311186 5954 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311193 5954 flags.go:64] FLAG: --feature-gates="" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311200 5954 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311205 5954 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311211 5954 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311216 5954 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311221 5954 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311234 5954 flags.go:64] FLAG: --help="false" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311239 5954 flags.go:64] FLAG: --hostname-override="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311245 5954 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311250 5954 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311255 5954 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311261 5954 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311267 5954 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311271 5954 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:57:39.319783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311276 5954 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311281 5954 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311285 5954 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311291 5954 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311296 5954 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311301 5954 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311306 5954 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311311 5954 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311316 5954 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311320 5954 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311325 5954 flags.go:64] FLAG: --lock-file="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311330 5954 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311335 5954 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311340 5954 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311349 5954 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311354 5954 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311359 5954 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311364 5954 flags.go:64] FLAG: --logging-format="text" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311371 5954 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311377 5954 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311382 5954 flags.go:64] FLAG: --manifest-url="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311387 5954 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311394 5954 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311399 5954 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311405 5954 flags.go:64] FLAG: --max-pods="110" Nov 03 19:57:39.320399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311420 5954 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311426 5954 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311431 5954 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311436 5954 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311441 5954 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311446 5954 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311451 5954 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311462 5954 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311468 5954 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311473 5954 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311478 5954 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311483 5954 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311488 5954 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311492 5954 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311497 5954 flags.go:64] FLAG: --port="10250" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311502 5954 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311506 5954 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08639579d593024cd" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311512 5954 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311517 5954 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311522 5954 flags.go:64] FLAG: --register-node="true" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311526 5954 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311533 5954 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311537 5954 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311542 5954 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311546 5954 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311555 5954 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:57:39.321003 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311561 5954 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311566 5954 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311571 5954 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311576 5954 flags.go:64] FLAG: --runonce="false" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311580 5954 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311585 5954 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311590 5954 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311603 5954 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311610 5954 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311615 5954 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311620 5954 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311626 5954 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311631 5954 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311635 5954 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311640 5954 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311645 5954 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311650 5954 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311655 5954 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311660 5954 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311669 5954 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311674 5954 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311679 5954 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311685 5954 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311690 5954 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311695 5954 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:57:39.321728 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311699 5954 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311704 5954 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311709 5954 flags.go:64] FLAG: --v="2" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311717 5954 flags.go:64] FLAG: --version="false" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311724 5954 flags.go:64] FLAG: --vmodule="" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311730 5954 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.311737 5954 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311963 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311970 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311976 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311980 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311985 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311991 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.311998 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312005 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312013 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312017 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312022 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312026 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312031 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312035 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.322404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312039 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312044 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312048 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312053 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312057 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312061 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312065 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312070 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312075 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312079 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312084 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312088 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312095 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312118 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312124 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312129 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312134 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312141 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312145 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.322926 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312150 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312155 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312159 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312163 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312168 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312172 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312177 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312183 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312189 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312194 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312198 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312202 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312207 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312211 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312216 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312220 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312224 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312229 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312233 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312238 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.323404 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312242 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312247 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312251 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312255 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312260 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312264 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312269 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312273 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312277 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312281 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312291 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312295 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312300 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312304 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312309 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312314 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312318 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312322 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312327 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.323902 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312333 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312338 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312343 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312348 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312352 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312357 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312361 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312366 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312370 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312375 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312379 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312383 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312389 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312393 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312397 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312402 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312406 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312411 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312415 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312419 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.324483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312424 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312429 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312433 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312439 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312444 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312448 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312452 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312456 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312459 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312463 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312468 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312472 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312479 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312484 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.312534 5954 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312598 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.325001 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312605 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312611 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312615 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312620 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312624 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312628 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312633 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312637 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312642 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312649 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312655 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312661 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312665 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312670 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312674 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312679 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312683 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312688 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312692 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.325467 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312697 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312702 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312706 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312711 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312722 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312726 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312731 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312736 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312740 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312745 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312753 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312760 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312765 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312771 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312775 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312780 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312785 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312789 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312794 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.325942 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312798 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312803 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312807 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312811 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312816 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312820 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312825 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312829 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312833 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312838 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312843 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312847 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312851 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312856 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312860 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312865 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312869 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312874 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312878 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312883 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.326495 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312887 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312891 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312896 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312902 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312906 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312910 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312915 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312920 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312925 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312929 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312934 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312938 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312943 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312948 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312952 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312957 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312961 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312966 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312970 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.326983 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312975 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312980 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312984 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312988 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312993 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.312997 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313001 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313006 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313010 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313015 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313020 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313024 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313028 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313033 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313037 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313041 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313048 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313052 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313057 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313061 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.327483 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313065 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313070 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313076 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313080 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313085 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313089 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313094 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.313098 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.318406 5954 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.318426 5954 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.318441 5954 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.318447 5954 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318509 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318514 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318517 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318520 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318524 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318526 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318529 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318532 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.328209 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318535 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318538 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318540 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318543 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318546 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318548 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318551 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318554 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318556 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318559 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318562 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318565 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318567 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318571 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318574 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318577 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318580 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318582 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318590 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318593 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.328803 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318596 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318599 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318601 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318604 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318607 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318609 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318612 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318615 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318617 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318620 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318623 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318626 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318630 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318634 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318638 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318642 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318646 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318649 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318653 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.329334 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318656 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318659 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318662 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318666 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318669 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318672 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318675 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318678 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318681 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318683 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318686 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318689 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318691 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318694 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318697 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318700 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318703 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318706 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318708 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.329797 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318711 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318714 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318716 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318719 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318722 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318725 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318728 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318730 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318733 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318736 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318739 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318741 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318744 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318747 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318750 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318753 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318756 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318759 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318761 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318764 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.330329 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318767 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318769 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318772 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318775 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318777 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318780 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318783 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318785 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318788 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318790 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318793 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318796 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318799 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318801 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318804 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318807 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318810 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318813 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318815 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.330812 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.318818 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.318845 5954 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319021 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319026 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319030 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319033 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319036 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319039 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319043 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319046 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319049 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319051 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319054 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319057 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319060 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319063 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.331351 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319065 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319068 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319071 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319073 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319076 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319078 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319081 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319084 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319087 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319090 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319093 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319095 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319098 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319100 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319115 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319118 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319120 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319123 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319126 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319129 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.331985 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319132 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319136 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319141 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319144 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319147 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319150 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319153 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319156 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319159 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319162 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319165 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319167 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319170 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319173 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319175 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319178 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319181 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319184 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319187 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.332855 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319189 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319192 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319195 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319198 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319201 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319203 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319206 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319208 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319211 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319213 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319216 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319218 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319221 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319224 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319226 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319229 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319232 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319235 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319239 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319242 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.333549 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319245 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319248 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319250 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319253 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319256 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319258 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319261 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319264 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319266 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319269 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319272 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319274 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319277 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319280 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319282 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319285 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319288 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319291 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319293 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319296 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.334042 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319298 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319301 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319304 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319306 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319309 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319312 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319314 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319317 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319321 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319324 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319328 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319332 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319334 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.319359 5954 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319393 5954 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:39.334679 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319398 5954 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319401 5954 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319404 5954 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319407 5954 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319409 5954 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319412 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319415 5954 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319417 5954 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319420 5954 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319423 5954 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319425 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319428 5954 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319431 5954 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319435 5954 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319439 5954 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319442 5954 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319445 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319447 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319450 5954 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:39.335088 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319452 5954 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319455 5954 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319457 5954 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319460 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319463 5954 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319465 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319468 5954 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319471 5954 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319473 5954 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319476 5954 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319479 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319481 5954 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319484 5954 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319487 5954 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319490 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319492 5954 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319495 5954 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319497 5954 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319500 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319502 5954 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:39.335571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319505 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319508 5954 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319510 5954 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319513 5954 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319515 5954 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319518 5954 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319520 5954 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319523 5954 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319525 5954 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319528 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319530 5954 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319533 5954 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319536 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319538 5954 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319542 5954 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319546 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319548 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319551 5954 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319554 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:39.336054 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319556 5954 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319559 5954 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319562 5954 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319564 5954 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319567 5954 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319574 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319577 5954 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319579 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319582 5954 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319585 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319588 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319591 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319594 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319597 5954 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319600 5954 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319602 5954 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319605 5954 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319608 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319611 5954 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319613 5954 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:39.336568 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319616 5954 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319618 5954 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319621 5954 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319624 5954 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319626 5954 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319629 5954 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319631 5954 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319634 5954 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319637 5954 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319639 5954 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319642 5954 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319645 5954 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319648 5954 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319652 5954 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319655 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319658 5954 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319660 5954 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319665 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319668 5954 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319671 5954 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:39.337064 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319673 5954 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319676 5954 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319679 5954 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319681 5954 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319684 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319686 5954 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:57:39.319689 5954 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.319750 5954 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.320782 5954 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.320851 5954 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.321036 5954 server.go:1014] "Starting client certificate rotation" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.321151 5954 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.321214 5954 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:28 +0000 UTC" deadline="2028-07-29 07:28:51.550236984 +0000 UTC" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.321255 5954 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15203h31m12.228985809s" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.321539 5954 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.323866 5954 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:39.337571 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.325431 5954 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:57:39.337968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.327579 5954 log.go:25] "Validated CRI v1 image API" Nov 03 19:57:39.337968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.328411 5954 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:57:39.337968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.331025 5954 fs.go:135] Filesystem UUIDs: map[473aad34-b4bc-48e8-bd6b-9d19cf305d57:/dev/nvme0n1p4 4f429464-8a0b-44ee-b611-69c67c790439:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 19:57:39.338073 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.331045 5954 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7/userdata/shm major:0 minor:441 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afc10babb45756f066b43615728140b72b64ee217c3b75f59b2c86f5ad4befd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afc10babb45756f066b43615728140b72b64ee217c3b75f59b2c86f5ad4befd4/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be/userdata/shm major:0 minor:482 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0/userdata/shm major:0 minor:478 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~projected/kube-api-access-98k5c:{mountpoint:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~projected/kube-api-access-98k5c major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/alertmanager/9 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/config-reloader/6 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~empty-dir/config-out major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/kube-api-access-846qk:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/kube-api-access-846qk major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/tls-assets major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/config-volume major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/web-config major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18e97b6a-be41-42af-90ec-bd56e6d791d6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/18e97b6a-be41-42af-90ec-bd56e6d791d6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216dd34c-6ab8-4044-97d2-d2ed537c6153/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/216dd34c-6ab8-4044-97d2-d2ed537c6153/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~projected/kube-api-access-rzlsx:{mountpoint:/var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~projected/kube-api-access-rzlsx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~projected/kube-api-access-bg24s:{mountpoint:/var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~projected/kube-api-access-bg24s major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~projected/kube-api-access-9hkqf:{mountpoint:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~projected/kube-api-access-9hkqf major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32c657c7-cf56-4ca8-b035-fd7c55fc2480/volumes/kubernetes.io~projected/kube-api-access-mjlc9:{mountpoint:/var/lib/kubelet/pods/32c657c7-cf56-4ca8-b035-fd7c55fc2480/volumes/kubernetes.io~projected/kube-api-access-mjlc9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3371a194-9a9e-4a3a-9af3-48d86152b1dc/volumes/kubernetes.io~projected/kube-api-access-gzfzx:{mountpoint:/var/lib/kubelet/pods/3371a194-9a9e-4a3a-9af3-48d86152b1dc/volumes/kubernetes.io~projected/kube-api-access-gzfzx major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~projected/kube-api-access-khsbl:{mountpoint:/var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~projected/kube-api-access-khsbl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~projected/kube-api-access-28x8q:{mountpoint:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~projected/kube-api-access-28x8q major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~projected/kube-api-access-kpxjr:{mountpoint:/var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~projected/kube-api-access-kpxjr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~secret/metrics-certs major:0 minor:469 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47f13c13-8d81-472c-8a39-2f14c8da0242/volumes/kubernetes.io~projected/kube-api-access-4gfwm:{mountpoint:/var/lib/kubelet/pods/47f13c13-8d81-472c-8a39-2f14c8da0242/volumes/kubernetes.io~projected/kube-api-access-4gfwm major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~projected/kube-api-access-7h9jj:{mountpoint:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~projected/kube-api-access-7h9jj major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~projected/kube-api-access-q554h:{mountpoint:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~projected/kube-api-access-q554h major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~projected/kube-api-access-fh7nl:{mountpoint:/var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~projected/kube-api-access-fh7nl major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~projected/kube-api-access-q5zkt:{mountpoint:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~projected/kube-api-access-q5zkt major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/945ef139-c1e8-4154-b869-f18adcc62d8a/volumes/kubernetes.io~projected/kube-api-access-pgr8h:{mountpoint:/var/lib/kubelet/pods/945ef139-c1e8-4154-b869-f18adcc62d8a/volumes/kubernetes.io~projected/kube-api-access-pgr8h major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9edf52cd-afe4-46cb-b9b3-148d5445ca6e/volumes/kubernetes.io~projected/kube-api-access-qcl46:{mountpoint:/var/lib/kubelet/pods/9edf52cd-afe4-46cb-b9b3-148d5445ca6e/volumes/kubernetes.io~projected/kube-api-access-qcl46 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~projected/kube-api-access-qfnh2:{mountpoint:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~projected/kube-api-access-qfnh2 major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/prometheus/15 major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~empty-dir/config-out major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/kube-api-access-jspk2:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/kube-api-access-jspk2 major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/tls-assets major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/config major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/web-config major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014/volumes/kubernetes.io~projected/kube-api-access-pxt22:{mountpoint:/var/lib/kubelet/pods/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014/volumes/kubernetes.io~projected/kube-api-access-pxt22 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8/volumes/kubernetes.io~projected/kube-api-access-q6mqs:{mountpoint:/var/lib/kubelet/pods/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8/volumes/kubernetes.io~projected/kube-api-access-q6mqs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~projected/kube-api-access-9gnwr:{mountpoint:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~projected/kube-api-access-9gnwr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~projected/kube-api-access-blbkp:{mountpoint:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~projected/kube-api-access-blbkp major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~projected/kube-api-access-z77n8:{mountpoint:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~projected/kube-api-access-z77n8 major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:289 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ea0c82b4918e9537226f8c6e330687e576c35d0aab1ce3427f163136ed180da8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/366a89f6f3c32b38c7827ccfa4980042f4f65c3be1f6512bfea633f847bc3fec/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3086a33f4b91ea9d43adb2c14b203ab2749b878322ca6db89fe391abbb40dd7a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2d90a457861221b94f8acc0df59d744ef378ab694d0ecc329c7397777441f06e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/43067bce0e02ec2bddceab18943e1b18e1b9d2623ebfab20cd309816e474e320/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/044337d89055f661029d1630568b5356ce15b049a2209ade32cb1a948f8fbf9e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1c473c84160970fb07e3dc26b7a60e75c7d0c595276abbd25881af1083348e86/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6f85a59d4fe833f863105cc3f50c02db725e4ca7afd71b472240af70172dfc3b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a356edc2ca70d806e4787b22aaa9654f8a113537be9b38be49d6a270968a3f01/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/4d846f55c99f95307f745430a6abd15b4fa27fd2d1966d48276d95b8d4dd5958/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/ca9d41b1f448266ea4c389c485b7475b3ec8779b2ae08f393a0e71b313d19d0a/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1503ba5d8150abb7f917eb1f27da7c6e4788155fd2dac4d559f454a14faa378a/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a72e2804dbd93d512013ce4badd082f49814c6736540eb3b1083e41ce12c113c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fc3111a14b29a972e69cb4dd94ec18e517353de8a7a1cc82fcbd69d9d53c5f34/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d7b9398f1d959425c5014f3579c8fd55c9b09415b0910dfa3f3907ae8d5504e8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2c8f720e2b8aeefeaf30ffbdfb2b77a690e8fa221e4d14852b3cf2a85101a6d1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4218b29da3e9c7f60645f782d4387f20af7446e2d8ddab2537c9eb214c29df93/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2be66c29bd0d232b1bfd2513430d611d8a9db52cfadea5c8a2471aeb2f4f15ef/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bd5be95e2b0a49f4f482be6a72e7e6ce64457a205071f4a1fe84b4dab3d54acf/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f4ec82e5e10f0e59653f6a15a6f2ae4d07f28f614498bcbdbb957e27db2d0146/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a23a95e6857a1e8273a9c31a5ab534c23c81a25eb0a837e961ffae59663b4b21/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/0e489432675a07457bbf6278c58f41426b74a3dfd1a108ceae3668fc506be5a5/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a3c99f99f290a5b738ba636cfbbf65a7cbf0a9d52dd465daefc2472dac369c4e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/a5e59d181b96cf2c56a9f4f974e4a5d65914ee602ba0f8229e545070f10467d5/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/e4660e65926e9df5312016ce706a538a4170b959898146779ea773beb4eb8113/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/0d475272ea8cc8b4e42dec14d36106b3cd6f8d24d67b99458ac7906fc3f87007/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/24dca1db97c4d5b03d5001e9abcb31caca83b1a632440f8cfbe25348d2468561/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/286b67130831791f6e76315600cd4de34d8848f284eba92336406bd36d98e2ce/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/289effddd7eabc624dbfabfaa17f4bd7caae18cbd3ba4080e75aacfb9f19147e/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/6032279e06fd7af9a84089c4a6c9762ab15305735aebf867e49c420a2ec25de0/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/0fcfd1957e417abd1388a9a5f11d164845d4b0a17965798c2ba94c43dde3bf70/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/00afca3b110a6a60c3a2dcfec5c40378064b055b2ab76d6de5daced819f8d7b3/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/3cd6578e910329aca4d820a26aea51d0336d8bfbea85846edc8f4919c22a4bb6/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/7203ec6978bd058cda86321ae1abcfb35b54d880663ff0c5d67d4312bee3f53f/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/841604fae4dbf919bb4e6ef5c2a55edaacd504ef491cc926b3a9399071026f77/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/99999b9cd8b616a090ba9efae6fc9faa761ae340d775fee57b2c98abff308f98/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/c41f1da6b9f09a6ffccb42433d44ea358406e3d54678c7036c07ddd4d39d42d0/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/a121ea3474dbfc10368544d43e05e1dafcf3f22e774d8c926714bfe136c02753/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/7d514e53ff73ef19c63b4cb40c6af310ed481a2d1ce3ac1d72079253448d0135/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/e55193fed17daab3b3dbc8a4b1e2485e5d2fa512aa25d762f668d8c5217f7617/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/7724ca87d9356850676555cf3c1d399f4093d006db64ca8b168be0def95659aa/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/5f2be5802eea2e5fe4f75fc8644a99e09b7a6264ad17c4928e7251af5a80b9fc/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/2ce1f840e112d766d0c011bd300faf9385759973089b667363d97c1cb4c5abae/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/5b7b8253f907209028ec30b02d622101ea03bea3ff5520e0dfd1e0d23fabd294/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/b5faa62a43859486aff49f4d6346bc3c03d0d83967d2272577bc0eccce3b640c/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/990a939aa54f09eebfc3f83972d4a464a16b65e4fbf6a0b945e9bf79a38aac82/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/f84a388e56d8f381f54b81c55e1db47e6736b9d57bb00b40902512eff3154185/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/a5d3593e47f9fe2d5d567614146087f675d57367e465e392b2f5d84b4bb8eb5e/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/c39a394460637dc1045cfa80709ef5710106756d17da10dca4fc3b09a4e6e0ec/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/efcebfd763fbf21b6a15f167ab92c6b68b0ceec6101762eca1cb0af5a95c61c5/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/b1ae8e44970936669f79a75a0517c3dca5e6730d7d3dfe86d4e37387b0178c0b/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/f32dd67db05fd713bef9ac235ec53125fb2cbac8ea40c484beff3be8aa067acb/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/0fee65261e9f4e59e7f5f34cc2b3e361eb4ffa455e86330782ed7ac2cc2390a3/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/440eb77082fde884f20bc28a7a8c9ddbcd7f14edac64f8f37900700f3b94e05f/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/2663abd39f80a0d7c1a13a38e3b766499bfc58cd2e946eec00cd763c738aeeeb/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/faae6e85fb78a54cc323300a8132b52f44e4985ea2d3ff641c17cd996b6eea47/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/4b6bed7c05f0937ae8ced59cf7b0773c404aa42f912596b3beca5cc4e59ab253/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/cda8b13d53c094f2e7559a3f0664ce7a376eef2b0ccd26156a3d3042cd8aa757/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/d79f5d331a432be964c9313409ebd36be8881f191430ce2678761467416a2d3d/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/187af44c1ed1f84067910d6a16a6974de05067720f21259532e84b1a5d221270/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/b91d1696cb324ac07876057b4065eb5a64f326ba65ecd0ac7cd8d432d6f8a7a6/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/8581353e21962e96e23e3126d131617e9faeb216d6fc26fb176ff1957871d71a/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/e93c401fd77240bc4c36da0d45282da6735fccb3c5bad65f874c3a59f4e10efb/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/6ad32b9eb28b908deb364357f6eb8884f3f42e9de0e8fea097022949fab4e9c6/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/c9d7417379c001976341faf09cd57bc92271ddf8bb157b9f26ec333e6a3bc3a0/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/c0a1137ce3e65c9ae496a458a87d76c4f7208b6f1923acc9efeacc6013cedf6e/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/6060a8dcadebb6a9fce2a4aba6fa353d40abbf158b9b6272df9088720e878641/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/47f493a4623f9e5de215e7da45a717645482a3e40f14ab84c5a2dd0cd758002e/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/48855c8635c4212a7801d4af2807681cbf21ba164adb6312ff39f831d9f1a7e3/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/ab5fb71bac7e6d82a3a20807f63b3dbee435dbfc55d9c66ad465082be90ed039/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/596ee543151a2ca5175bce24046001d6220ac182383de2a76a6a89975754225b/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/2def39b14c6fda0bacc15297b65ef0812cc4853de3324be5d8e096e612703d94/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/3c0113bb62eb7526a9ba444c5f42f29430e68b85e7f81a59c2b28c5af135c5bf/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/951a5682394d0693b03c038b9fe2fec5f330763c302a45303debba7f7a8e85ba/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/11b667656f1719946bf761931dcf1260c638be25beaa5e5cda4527ae629c3248/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/f0c66ed4299892c44d377814d1f0c6d6aa6694db6501d8cb359172b711ffa2ad/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a16422ebde865a8efd7157d967b86d226a5c7c2c6caaa7ecda20b61a08326d66/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/71138739aa8ee994c4a9b25dfe642b4867f5367132984689b86d592772570dc7/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6952efdd6ab75fbba32e8dbf3213e9ef39b3612a97915dc033083710b600e71a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1a008afa69e0541fed6a0fcee0938b694a96c0a5f0104bd5fc5f4c41836e69f9/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/88f80a32c6099fa0eebea191dc031b0b5549cdca1292983f93c4197fba79de94/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3f6676fb10f45a2717d7e264c134d4576742c72bf3b88418291c7299f2e5d33d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/efc26d3d24e62b955f930c207923773f53cd3085ace30534a308b3a2aaf1ff91/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f295fd859ecf48aac158ddf8f6a7dccd7bdd12226cc5ab83569aa6ba38d8b828/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e3c1c826a2426516c2a64c19af8dc337a76bbeb97225959918ebf270652d96f4/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:57:39.342441 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.341897 5954 manager.go:217] Machine: {Timestamp:2026-11-03 19:57:39.34139506 +0000 UTC m=+0.066402648 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099732 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28b95a672c39aab484d22307f0a736 SystemUUID:ec28b95a-672c-39aa-b484-d22307f0a736 BootID:6ee72659-cb27-42c1-bb1c-d4a23aef1e75 Filesystems:[{Device:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~projected/kube-api-access-qfnh2 DeviceMajor:0 DeviceMinor:409 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~projected/kube-api-access-q5zkt DeviceMajor:0 DeviceMinor:392 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:432 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c832a3c7278f3b85a06c5cd52f56984b5364f6d4211da5e1356a8ba0b28c99be/userdata/shm DeviceMajor:0 DeviceMinor:482 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf82b71489944b6e55b2906a8d03dfdfd1c3b58fdcc5b97c510267c5864aae61/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~projected/kube-api-access-rzlsx DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caedf99cfed472a2610d6fe15adf583e9c5b106913c1b27a2071154e8e3401d0/userdata/shm DeviceMajor:0 DeviceMinor:478 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3371a194-9a9e-4a3a-9af3-48d86152b1dc/volumes/kubernetes.io~projected/kube-api-access-gzfzx DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7/userdata/shm DeviceMajor:0 DeviceMinor:441 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~projected/kube-api-access-q554h DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa0650af8ac1dce349877ca09cb6aa0f9a40749f8d13f22ce5289a19ec949f7a/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~projected/kube-api-access-kpxjr DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:423 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/kube-api-access-jspk2 DeviceMajor:0 DeviceMinor:440 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~projected/kube-api-access-blbkp DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:260 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/945ef139-c1e8-4154-b869-f18adcc62d8a/volumes/kubernetes.io~projected/kube-api-access-pgr8h DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014/volumes/kubernetes.io~projected/kube-api-access-pxt22 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efd31960e48614da5143bd5b44d0a6b14406196ee0b775ce0408eb0c524a94ef/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:333 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:433 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:436 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:405 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:388 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~projected/kube-api-access-98k5c DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ba4644c5cf0680bc947b1777fb5317ed5ac0c20bd1df678e59cd3ce96c8479e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:434 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8/volumes/kubernetes.io~projected/kube-api-access-q6mqs DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f3b597639b047b44f6c9272bccda2fb999de6bcce40f725e39a45c4734e5b97/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~projected/kube-api-access-9hkqf DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cea4cd566a8623303f2dc2e801d9f89e437cf05ee1f9efdccd21816047c796ea/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~projected/kube-api-access-fh7nl DeviceMajor:0 DeviceMinor:408 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06728ec3-24de-459c-b966-8b88db3bc248/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c73db5acee3a9b02609f1569757190521b5d83e52c35137f2dafab200b1a13a4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~projected/kube-api-access-bg24s DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afc10babb45756f066b43615728140b72b64ee217c3b75f59b2c86f5ad4befd4/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5b6773e60acbdba6a41db9b2564451087224e05b19177de982e1a1d0eeef8d7/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc3cd7f2-0667-4487-965c-44e36b07ceb3/volumes/kubernetes.io~projected/kube-api-access-z77n8 DeviceMajor:0 DeviceMinor:292 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:439 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~projected/kube-api-access-9gnwr DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/volumes/kubernetes.io~projected/kube-api-access-khsbl DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4565b7eeb6fafacbed1fe26d9190d454ff662b0815c4e142b55052a241bf2599/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72baa86116bfb75a63ac3b5c37a2ef25cfcd006104faa35c20b3ded8b4ab6a6b/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:438 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e98678215a495f271a1a1fd22b19ad33e7bf48f18639fa6c4ed7786b6d62e749/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:422 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes/kubernetes.io~projected/kube-api-access-7h9jj DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:435 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07e97c862385d65c9a24d4f290b32d90b46d46a7a9c71b5a4c80624001669739/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/216dd34c-6ab8-4044-97d2-d2ed537c6153/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32c657c7-cf56-4ca8-b035-fd7c55fc2480/volumes/kubernetes.io~projected/kube-api-access-mjlc9 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f8fc60508ddcba9095850867f1a8e94d6f90e2bce6dd177f2065978e14ef445/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f969a4779b99ae1d76955ad958407710982a9ccdfb5f0e2066f110c5de62d097/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9edf52cd-afe4-46cb-b9b3-148d5445ca6e/volumes/kubernetes.io~projected/kube-api-access-qcl46 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1fcd9fc42e21f02f551592e865ec656f5c393b81403e2fe2315381f977db818/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/47f13c13-8d81-472c-8a39-2f14c8da0242/volumes/kubernetes.io~projected/kube-api-access-4gfwm DeviceMajor:0 DeviceMinor:470 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18e97b6a-be41-42af-90ec-bd56e6d791d6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:465 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e1076f-26b7-40eb-a11f-b9eca41ee6fc/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes/kubernetes.io~projected/kube-api-access-846qk DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73f24aac-5672-4b00-b199-324e3da68852/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:403 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2790512d-386a-41e8-8bb6-31ec8faafbc6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f0156c4609e63485f84821d70951acb59cc99d843b474e2111c7c8eb7884695/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~projected/kube-api-access-28x8q DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:384 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:431 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/426ca1a6-344a-4f33-9e58-50be798ed0a9/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:404 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/471a887a-a065-4aa3-9063-555b93a9a258/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:469 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7914fd824f2462e81a1a77609fb7440632caffaefdaf3a2f1b9ae4fb11cdf5e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fc34dba5-14d2-402c-99ed-0a27e607616d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/78dccc20-8871-4155-aa48-7ffbf940209f/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:391 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07e97c862385d65 MacAddress:ae:de:80:1b:49:d6 Speed:10000 Mtu:8901} {Name:0f3b597639b047b MacAddress:d2:cd:32:13:88:f9 Speed:10000 Mtu:8901} {Name:2d8a303339b3e56 MacAddress:a2:26:ff:0a:e6:41 Speed:10000 Mtu:8901} {Name:3291015c61d2ddc MacAddress:4e:0c:71:71:83:34 Speed:10000 Mtu:8901} {Name:4565b7eeb6fafac MacAddress:8a:c5:a0:4c:1e:8b Speed:10000 Mtu:8901} {Name:559dbacc8f6f9d1 MacAddress:7a:5a:51:25:f4:c8 Speed:10000 Mtu:8901} {Name:57784f1d0095db5 MacAddress:fe:9a:02:14:3f:98 Speed:10000 Mtu:8901} {Name:72baa86116bfb75 MacAddress:8a:85:45:b0:b4:6e Speed:10000 Mtu:8901} {Name:78ecdd1176e8349 MacAddress:ee:2e:50:29:bc:67 Speed:10000 Mtu:8901} {Name:7f0156c4609e634 MacAddress:06:49:cb:32:c0:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:57:1e:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:46:1f:8d:03:d5:41 Speed:0 Mtu:8901} {Name:c832a3c7278f3b8 MacAddress:ea:2e:98:8d:25:fc Speed:10000 Mtu:8901} {Name:caedf99cfed472a MacAddress:fa:62:c7:e7:fc:5b Speed:10000 Mtu:8901} {Name:e7914fd824f2462 MacAddress:1e:f4:71:d5:b8:44 Speed:10000 Mtu:8901} {Name:e98678215a495f2 MacAddress:5a:b5:13:e7:bc:e3 Speed:10000 Mtu:8901} {Name:efd31960e48614d MacAddress:e2:ab:55:69:82:18 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:57:1e:3f Speed:0 Mtu:9001} {Name:fa0650af8ac1dce MacAddress:22:e2:bd:6e:d2:33 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:6a:f7:13:43:76 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:84:0c:32:f2:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:57:39.342716 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342447 5954 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:57:39.342716 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342505 5954 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:57:39.342854 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342824 5954 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:57:39.342986 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342855 5954 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-70.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:57:39.343030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342989 5954 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:57:39.343030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.342997 5954 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:57:39.343030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343008 5954 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:57:39.343030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343025 5954 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:57:39.343162 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343157 5954 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:57:39.343352 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343342 5954 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:57:39.343458 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343450 5954 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:57:39.343493 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343488 5954 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:57:39.343522 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343502 5954 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:57:39.343522 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343513 5954 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:57:39.343522 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343522 5954 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:57:39.343608 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.343530 5954 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:57:39.344365 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.344331 5954 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:57:39.344365 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.344360 5954 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:57:39.344881 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.344862 5954 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:57:39.348000 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.347985 5954 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:57:39.348000 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.347999 5954 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:57:39.348098 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348089 5954 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348571 5954 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348622 5954 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348766 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348782 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348791 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348800 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348809 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348820 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348836 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348846 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348857 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348866 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348896 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348912 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348944 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:57:39.349736 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.348958 5954 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:57:39.352959 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.352940 5954 server.go:1263] "Started kubelet" Nov 03 19:57:39.353039 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.353014 5954 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:57:39.353654 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.353086 5954 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:57:39.353654 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.353204 5954 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:57:39.353654 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.353602 5954 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:57:39.353566 ip-10-0-135-70 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:57:39.357287 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.357270 5954 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:57:39.357459 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.357410 5954 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-70.us-east-2.compute.internal\" not found" Nov 03 19:57:39.357539 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.357509 5954 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358177 5954 factory.go:55] Registering systemd factory Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358201 5954 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.357568 5954 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.357555 5954 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358329 5954 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358490 5954 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:30 +0000 UTC" deadline="2028-05-16 16:37:38.492167189 +0000 UTC" Nov 03 19:57:39.358610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358512 5954 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13436h39m59.133658167s" Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358775 5954 factory.go:153] Registering CRI-O factory Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358792 5954 factory.go:223] Registration of the crio container factory successfully Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358848 5954 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358868 5954 factory.go:103] Registering Raw factory Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.358883 5954 manager.go:1199] Started watching for new ooms in manager Nov 03 19:57:39.360587 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.360010 5954 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:57:39.361007 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.360992 5954 manager.go:319] Starting recovery of all containers Nov 03 19:57:39.367060 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.366292 5954 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:39.367060 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.366426 5954 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:39.367937 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.367899 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log" seLinuxMountContext="" Nov 03 19:57:39.368032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.367949 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets" seLinuxMountContext="" Nov 03 19:57:39.368032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.367964 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3cd7f2-0667-4487-965c-44e36b07ceb3" volumeName="kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368568 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06728ec3-24de-459c-b966-8b88db3bc248" volumeName="kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368605 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368624 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368633 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368688 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6" volumeName="kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir" seLinuxMountContext="" Nov 03 19:57:39.368699 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.368701 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6" volumeName="kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369476 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369512 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369524 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471a887a-a065-4aa3-9063-555b93a9a258" volumeName="kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369532 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369542 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369555 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369565 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.369594 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369578 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369595 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c60f9fad-b8c0-4d6e-b755-c2bda3f2d014" volumeName="kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369635 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06728ec3-24de-459c-b966-8b88db3bc248" volumeName="kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369647 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369663 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" volumeName="kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369675 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369689 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369702 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369717 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369728 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369740 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369751 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369768 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c657c7-cf56-4ca8-b035-fd7c55fc2480" volumeName="kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369780 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945ef139-c1e8-4154-b869-f18adcc62d8a" volumeName="kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369792 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" volumeName="kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369806 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2790512d-386a-41e8-8bb6-31ec8faafbc6" volumeName="kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369817 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369831 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369844 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc34dba5-14d2-402c-99ed-0a27e607616d" volumeName="kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369857 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369872 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc34dba5-14d2-402c-99ed-0a27e607616d" volumeName="kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369887 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369909 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18e97b6a-be41-42af-90ec-bd56e6d791d6" volumeName="kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369925 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2790512d-386a-41e8-8bb6-31ec8faafbc6" volumeName="kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369936 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369959 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369974 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369984 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.369997 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370011 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config" seLinuxMountContext="" Nov 03 19:57:39.370017 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370029 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370043 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370059 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e458f2d3-a88a-44b2-8bd9-57bbb097bfa8" volumeName="kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370084 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f8e8c2-cae1-4e2a-92e0-427d4c4f4186" volumeName="kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370096 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370127 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f24aac-5672-4b00-b199-324e3da68852" volumeName="kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370140 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73f24aac-5672-4b00-b199-324e3da68852" volumeName="kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370152 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370165 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4cbdc4a-e9f3-4489-bd72-341746fdc0c9" volumeName="kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370177 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3cd7f2-0667-4487-965c-44e36b07ceb3" volumeName="kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370204 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370219 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370245 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370258 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2790512d-386a-41e8-8bb6-31ec8faafbc6" volumeName="kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370273 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6" volumeName="kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370285 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370297 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216dd34c-6ab8-4044-97d2-d2ed537c6153" volumeName="kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370311 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370323 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370336 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370350 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370362 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370373 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3cd7f2-0667-4487-965c-44e36b07ceb3" volumeName="kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370384 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06728ec3-24de-459c-b966-8b88db3bc248" volumeName="kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370396 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" volumeName="kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370399 5954 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370407 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc34dba5-14d2-402c-99ed-0a27e607616d" volumeName="kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370420 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc3cd7f2-0667-4487-965c-44e36b07ceb3" volumeName="kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370433 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370445 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc34dba5-14d2-402c-99ed-0a27e607616d" volumeName="kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370456 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370467 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216dd34c-6ab8-4044-97d2-d2ed537c6153" volumeName="kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370479 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f8e8c2-cae1-4e2a-92e0-427d4c4f4186" volumeName="kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370491 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c657c7-cf56-4ca8-b035-fd7c55fc2480" volumeName="kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370504 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370517 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="945ef139-c1e8-4154-b869-f18adcc62d8a" volumeName="kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370529 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4cbdc4a-e9f3-4489-bd72-341746fdc0c9" volumeName="kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370541 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370553 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370568 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" volumeName="kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370579 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370591 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370604 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e1076f-26b7-40eb-a11f-b9eca41ee6fc" volumeName="kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370615 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370660 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370676 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370689 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370703 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370734 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47f13c13-8d81-472c-8a39-2f14c8da0242" volumeName="kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370747 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06728ec3-24de-459c-b966-8b88db3bc248" volumeName="kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370759 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370772 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32c657c7-cf56-4ca8-b035-fd7c55fc2480" volumeName="kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370786 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3371a194-9a9e-4a3a-9af3-48d86152b1dc" volumeName="kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370802 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370820 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370831 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4cbdc4a-e9f3-4489-bd72-341746fdc0c9" volumeName="kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370843 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06728ec3-24de-459c-b966-8b88db3bc248" volumeName="kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370856 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6" volumeName="kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370869 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471a887a-a065-4aa3-9063-555b93a9a258" volumeName="kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370882 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370894 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370906 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370918 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="426ca1a6-344a-4f33-9e58-50be798ed0a9" volumeName="kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370931 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370943 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370982 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea0e226-87c0-40c9-b1b3-6029f76e6576" volumeName="kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.370996 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0ac0444-b127-4046-80b2-49f8d1a416e5" volumeName="kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371032 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371048 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2790512d-386a-41e8-8bb6-31ec8faafbc6" volumeName="kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371065 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371077 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371093 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c60f9fad-b8c0-4d6e-b755-c2bda3f2d014" volumeName="kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371119 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371131 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78dccc20-8871-4155-aa48-7ffbf940209f" volumeName="kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371142 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad270548-2169-4822-8c43-17f547dc0421" volumeName="kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371156 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf7348-9f76-4a5c-889a-4f687fe056b2" volumeName="kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371170 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3371a194-9a9e-4a3a-9af3-48d86152b1dc" volumeName="kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371184 5954 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488637a9-aaf1-4506-a7ad-5d20665bb414" volumeName="kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config" seLinuxMountContext="" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371197 5954 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:57:39.372334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.371205 5954 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:57:39.378684 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.377722 5954 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:57:39.395035 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.394998 5954 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:57:39.396706 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.396661 5954 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:57:39.396706 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.396699 5954 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:57:39.397316 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.396733 5954 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:57:39.397316 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.396777 5954 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:57:39.400492 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.400258 5954 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:39.408615 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.408581 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="dadada5ebba3a4035f3e88d8407fac4175aca7617af99b5be01b2a474e8fb945" exitCode=0 Nov 03 19:57:39.439926 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.439893 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" exitCode=0 Nov 03 19:57:39.449213 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.449185 5954 generic.go:356] "Generic (PLEG): container finished" podID="44458e05214beace7a3b673d5f048430" containerID="34a096f4cfa93c9576f7f046822e0132ef6896c923e9c566b7bf3eaf919e6cd3" exitCode=0 Nov 03 19:57:39.455196 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.455060 5954 generic.go:356] "Generic (PLEG): container finished" podID="426ca1a6-344a-4f33-9e58-50be798ed0a9" containerID="7cc4d081bbb479ff20f803c5ea60e02e195a163400f294d8c5386fb5cf4a9721" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464135 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="61940c346c408871137491f77e656130837b059388ac414303b8385608edcc1f" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464161 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="458d953385345ea007c281a3d488d0f5f1231a7bd1b28204dd5e0b0e38ec02b4" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464172 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="23483a5bb979ba1884c61ad0bc25eae681091b5f8892697622ec62f8c8c07b43" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464181 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="a9cc1ed23dda5efdb4235e6686ee07308c9fb8bdc368a238adc0a5b4beeb3f59" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464190 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="f847ccae830132cf852c8079bb8885ca4a5c1604a8c71c5dd9357b8ce889e48f" exitCode=0 Nov 03 19:57:39.464201 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.464197 5954 generic.go:356] "Generic (PLEG): container finished" podID="9edf52cd-afe4-46cb-b9b3-148d5445ca6e" containerID="d9fd02d8a7100347fd3d65a56aeaaf38cd17b03edb7a0ef36e7c90421693112b" exitCode=0 Nov 03 19:57:39.489617 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.489594 5954 manager.go:324] Recovery completed Nov 03 19:57:39.496926 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.496904 5954 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 19:57:39.515798 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515766 5954 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:57:39.515798 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515786 5954 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:57:39.515955 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515811 5954 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:57:39.515986 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515951 5954 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:57:39.515986 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515963 5954 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:57:39.515986 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515981 5954 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:57:39.516084 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515986 5954 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:57:39.516084 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.515992 5954 policy_none.go:50] "Start" Nov 03 19:57:39.516084 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.516002 5954 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:57:39.516084 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.516012 5954 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:57:39.516212 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.516152 5954 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:57:39.516212 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.516160 5954 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:57:39.516212 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.516165 5954 policy_none.go:44] "Start" Nov 03 19:57:39.521056 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.521023 5954 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:57:39.523710 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.523695 5954 manager.go:341] "Starting Device Plugin manager" Nov 03 19:57:39.524331 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.523722 5954 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:57:39.524369 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524335 5954 server.go:87] "Starting device plugin registration server" Nov 03 19:57:39.524610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524598 5954 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:57:39.524639 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524613 5954 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:57:39.524735 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524707 5954 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:57:39.524850 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524822 5954 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:57:39.524968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524863 5954 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:57:39.524968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.524870 5954 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:57:39.525432 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.525301 5954 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:57:39.524848263Z","UUID":"823402f9-17fb-4e37-be53-0365600c8bc4","Handler":null,"Name":"","Endpoint":""} Nov 03 19:57:39.527553 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.527514 5954 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:57:39.527970 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.527952 5954 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:57:39.528030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.527983 5954 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:57:39.624789 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.624752 5954 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:57:39.627067 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.627041 5954 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:57:39.627213 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.627077 5954 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:57:39.627213 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.627089 5954 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-70.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:57:39.627213 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.627152 5954 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.650516 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.650490 5954 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.650649 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.650566 5954 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.697389 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.697320 5954 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal"] Nov 03 19:57:39.698052 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698018 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"d161bc542803fe55ee71704a451b4d4d","Type":"ContainerStarted","Data":"63aa235581154c71d68d03b3d700d67a82109a9bde65cd375988bbcc1cddac4b"} Nov 03 19:57:39.698097 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698059 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"d161bc542803fe55ee71704a451b4d4d","Type":"ContainerStarted","Data":"947be900bef016f4f4ab65b1376ae38e4e6955f541825a1dfbe6380985e0b44a"} Nov 03 19:57:39.698171 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698101 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerStarted","Data":"eec22cdcda6c988e2748c66f56181755c4f73855d2f49e03c7bc73167ca4749c"} Nov 03 19:57:39.698171 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698145 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerDied","Data":"34a096f4cfa93c9576f7f046822e0132ef6896c923e9c566b7bf3eaf919e6cd3"} Nov 03 19:57:39.698171 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698161 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" event={"ID":"44458e05214beace7a3b673d5f048430","Type":"ContainerStarted","Data":"c672e3c4a29002816ef0863dc28ce658e0dacc541b3f9dbd519616397f671ad0"} Nov 03 19:57:39.698290 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698185 5954 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.698290 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.698191 5954 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.713443 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.713408 5954 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:39.713614 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.713502 5954 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.713614 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.713583 5954 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:39.713733 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:39.713656 5954 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.778973 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.778874 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.778973 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.778920 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.778973 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.778939 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.879933 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.879897 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.879933 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.879845 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.880137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.879964 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.880137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.879986 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.880137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.880035 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d161bc542803fe55ee71704a451b4d4d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"d161bc542803fe55ee71704a451b4d4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:39.880137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:39.880033 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44458e05214beace7a3b673d5f048430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" (UID: \"44458e05214beace7a3b673d5f048430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:40.344712 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.344662 5954 apiserver.go:52] "Watching apiserver" Nov 03 19:57:40.359047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.359021 5954 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:57:40.359814 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.359784 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-49krk","openshift-console/console-6b85776785-crvbv","openshift-dns/node-resolver-v2tvg","kube-system/global-pull-secret-syncer-lb594","kube-system/konnectivity-agent-8wq99","openshift-ovn-kubernetes/ovnkube-node-l9nr5","openshift-monitoring/monitoring-plugin-77f4f76594-52pfh","openshift-network-operator/iptables-alerter-tnxjb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d","openshift-monitoring/metrics-server-5687c6f496-d89k4","openshift-multus/multus-additional-cni-plugins-njwcg","openshift-multus/network-metrics-daemon-psmsw","kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal","openshift-monitoring/node-exporter-k7sbn","openshift-network-diagnostics/network-check-target-r97c6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw","openshift-console/console-6f46b5684f-zmphc","openshift-dns/dns-default-chgrg","openshift-ingress-canary/ingress-canary-8slb6","openshift-insights/insights-runtime-extractor-ltbwq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w","openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs","openshift-multus/multus-ndfvw","openshift-image-registry/node-ca-87bj5","openshift-monitoring/prometheus-k8s-0","openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:57:40.364240 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.364212 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:57:40.364375 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.364353 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.364430 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.364375 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.364502 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.364481 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.365836 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.365817 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:57:40.366437 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.366412 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:57:40.366644 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.366626 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:57:40.367246 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367225 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:57:40.367332 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367301 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:57:40.367500 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367485 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:57:40.367674 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367651 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:57:40.367853 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367833 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.367931 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367873 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:57:40.368014 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367999 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:57:40.368216 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368198 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.368281 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368253 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:57:40.368326 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368284 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:57:40.368460 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368446 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.368529 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368514 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.368623 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368605 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.368707 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368689 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.368826 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368809 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:57:40.368887 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368819 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.368932 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.367528 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:57:40.368975 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.368004 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:57:40.369024 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369006 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.369071 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369047 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:57:40.369071 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369053 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:57:40.369208 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369190 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.369255 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369223 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:57:40.369372 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369348 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.369497 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369373 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.369797 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369776 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:57:40.369887 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369872 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.370008 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.369993 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:57:40.370161 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370146 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:57:40.370222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370192 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:57:40.370222 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370200 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:57:40.370358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370341 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:57:40.370449 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370433 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:57:40.370502 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370470 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:57:40.370502 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370481 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:57:40.370595 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370518 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:57:40.370595 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370350 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:57:40.370595 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370573 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.370737 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370597 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:57:40.370737 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370602 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.370737 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370620 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:57:40.370737 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370705 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.370904 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370745 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.370904 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370813 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:57:40.370904 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.370834 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:57:40.371280 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.371262 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:57:40.372183 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.372159 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.372284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.372195 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:57:40.374758 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.374735 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 19:57:40.374965 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.374882 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.375050 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.374905 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fxkp4\"" Nov 03 19:57:40.375239 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.375222 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 19:57:40.375322 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.375253 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.375322 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.375318 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:57:40.378647 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.378618 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:57:40.460947 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.460909 5954 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:57:40.464369 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.464348 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:57:40.464498 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.464418 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 19:57:40.464555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.464516 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 19:57:40.464555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.464525 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:57:40.464641 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.464582 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:57:40.465328 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.465246 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:57:40.465328 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.465287 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 19:57:40.465328 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.465290 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 19:57:40.465784 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.465772 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:57:40.466553 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.466522 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:57:40.466655 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.466572 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 19:57:40.466655 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.466579 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:57:40.466974 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.466962 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 19:57:40.467326 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467311 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 19:57:40.467420 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467406 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 19:57:40.467463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467452 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 19:57:40.467507 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467495 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 19:57:40.467551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467505 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 19:57:40.467551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467540 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 19:57:40.467628 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467548 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 19:57:40.467668 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.467629 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 19:57:40.468015 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.468000 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 19:57:40.468142 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.468126 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:57:40.468237 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.468150 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:57:40.468237 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.468152 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:57:40.468237 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.468190 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:57:40.469495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.469474 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:57:40.469751 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.469733 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:57:40.469912 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.469897 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wbnrm\"" Nov 03 19:57:40.470015 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.469999 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:57:40.470232 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.470215 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 19:57:40.470403 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.470387 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fksvw\"" Nov 03 19:57:40.470453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.470429 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 19:57:40.470648 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.470621 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-z6gt5\"" Nov 03 19:57:40.470745 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.470662 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z5cft\"" Nov 03 19:57:40.471304 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471276 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bgwmm\"" Nov 03 19:57:40.471454 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471435 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:57:40.471554 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471538 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 19:57:40.471607 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471595 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mcwl7\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471639 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471651 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471749 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-648mvoststcbk\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471816 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471423 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471748 5954 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471302 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-llgd5\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471905 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjsd2\"" Nov 03 19:57:40.472032 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.471441 5954 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:40.472894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.472876 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 19:57:40.473368 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.473340 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f3423uefgu8jc\"" Nov 03 19:57:40.474597 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.474563 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 19:57:40.481960 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.481936 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.482054 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482010 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.482153 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482068 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.482153 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482115 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.482267 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482148 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.482267 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482219 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.482371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482262 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.482371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482270 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.482371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482291 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.482371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482330 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.482371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482357 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482436 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482486 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482525 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482536 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482568 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsbl\" (UniqueName: \"kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482575 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482587 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482613 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482622 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.482632 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482620 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482660 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482716 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482787 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482810 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482833 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-tmp-dir\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482838 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482882 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2790512d-386a-41e8-8bb6-31ec8faafbc6-data-volume\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482893 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482916 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482922 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482950 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482980 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483047 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.482982 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-script-lib\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483141 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483152 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483166 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483166 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483186 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483194 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483205 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483223 5954 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483230 5954 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483235 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77n8\" (UniqueName: \"kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483252 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:40.483261 5954 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483269 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483297 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483322 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483346 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483373 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98k5c\" (UniqueName: \"kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483364 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483403 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483271 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483439 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:57:40.483273 5954 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-70.us-east-2.compute.internal" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483475 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-tls\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.483555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483506 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483627 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483647 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh7nl\" (UniqueName: \"kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-metrics-client-ca\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2790512d-386a-41e8-8bb6-31ec8faafbc6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483749 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483817 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483828 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/216dd34c-6ab8-4044-97d2-d2ed537c6153-agent-certs\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483846 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-tmp\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483864 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483831 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-cert\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483892 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483918 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483938 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt22\" (UniqueName: \"kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483958 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483968 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.483982 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484012 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/471a887a-a065-4aa3-9063-555b93a9a258-metrics-certs\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484012 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484044 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484066 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484081 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484129 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484160 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484158 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484197 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484227 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484200 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/06728ec3-24de-459c-b966-8b88db3bc248-ovn-node-metrics-cert\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484274 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484283 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484379 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484395 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484445 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484471 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484475 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-metrics-server-audit-profiles\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484496 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484532 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlc9\" (UniqueName: \"kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484559 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.484590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484590 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73f24aac-5672-4b00-b199-324e3da68852-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484650 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484666 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484730 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484773 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484803 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484816 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484820 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484948 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484956 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484959 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc34dba5-14d2-402c-99ed-0a27e607616d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.484981 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485036 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485074 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485092 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485142 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485146 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-env-overrides\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485190 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzfzx\" (UniqueName: \"kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485220 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485253 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485264 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485254 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485323 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485355 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485382 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxjr\" (UniqueName: \"kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485327 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485405 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485405 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485451 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485460 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78dccc20-8871-4155-aa48-7ffbf940209f-audit-log\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485476 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485524 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnwr\" (UniqueName: \"kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485541 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485641 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485664 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485682 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485692 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485722 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485746 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485748 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485797 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485814 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485830 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485950 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485970 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485957 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.485985 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486027 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486086 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486148 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486177 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486188 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486200 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486228 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486375 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486380 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486422 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486450 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cni-binary-copy\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486457 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486502 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.486495 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486528 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486556 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486581 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486599 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486611 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkqf\" (UniqueName: \"kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486648 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486681 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486737 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-serviceca\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486741 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgr8h\" (UniqueName: \"kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486753 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-config-volume\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486769 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486778 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486796 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486822 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486849 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg24s\" (UniqueName: \"kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486873 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486888 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486927 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.486948 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487021 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487021 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487046 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487077 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487083 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487099 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487079 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487209 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487228 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487248 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487268 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487283 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487340 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/945ef139-c1e8-4154-b869-f18adcc62d8a-iptables-alerter-script\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487343 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487391 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487405 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487418 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487470 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487473 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487511 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mqs\" (UniqueName: \"kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487537 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487543 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-textfile\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487554 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487591 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487609 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487701 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487721 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487738 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-tuned\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487831 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487849 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcl46\" (UniqueName: \"kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487901 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487901 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487950 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487973 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/216dd34c-6ab8-4044-97d2-d2ed537c6153-konnectivity-ca\") pod \"konnectivity-agent-8wq99\" (UID: \"216dd34c-6ab8-4044-97d2-d2ed537c6153\") " pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.487990 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlsx\" (UniqueName: \"kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488023 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488093 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488179 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488141 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fc34dba5-14d2-402c-99ed-0a27e607616d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488210 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488239 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x8q\" (UniqueName: \"kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488247 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc3cd7f2-0667-4487-965c-44e36b07ceb3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-metrics-tls\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488265 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488290 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zkt\" (UniqueName: \"kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488294 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/06728ec3-24de-459c-b966-8b88db3bc248-ovnkube-config\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488316 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488359 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78dccc20-8871-4155-aa48-7ffbf940209f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488460 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488465 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488481 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488501 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blbkp\" (UniqueName: \"kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488512 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488524 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488632 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488658 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488662 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488675 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488704 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488835 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488864 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488866 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488886 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488910 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488917 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3371a194-9a9e-4a3a-9af3-48d86152b1dc-tmp-dir\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488971 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.488995 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489021 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489029 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489046 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489077 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489120 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489164 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489196 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489203 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489211 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489224 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489255 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489278 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-client-ca-bundle\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.489284 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489308 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489365 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489365 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489403 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489410 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489443 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489456 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489489 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489518 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489545 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489571 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489598 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489613 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489622 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489664 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489711 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489733 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489759 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489795 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489823 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489854 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489872 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e97b6a-be41-42af-90ec-bd56e6d791d6-original-pull-secret\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489879 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489901 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.489957 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-daemon-config\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.490100 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.490165 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.490170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc3cd7f2-0667-4487-965c-44e36b07ceb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.490170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-tls\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.490182 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78dccc20-8871-4155-aa48-7ffbf940209f-secret-metrics-server-client-certs\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.492517 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsbl\" (UniqueName: \"kubernetes.io/projected/400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6-kube-api-access-khsbl\") pod \"dns-default-chgrg\" (UID: \"400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6\") " pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.492705 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"console-6b85776785-crvbv\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:57:40.493191 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.492784 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.494363 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.494347 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77n8\" (UniqueName: \"kubernetes.io/projected/fc3cd7f2-0667-4487-965c-44e36b07ceb3-kube-api-access-z77n8\") pod \"openshift-state-metrics-7bd7489957-7cv4w\" (UID: \"fc3cd7f2-0667-4487-965c-44e36b07ceb3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7cv4w" Nov 03 19:57:40.496804 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.496777 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt22\" (UniqueName: \"kubernetes.io/projected/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-kube-api-access-pxt22\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.496884 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.496820 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"console-6f46b5684f-zmphc\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:57:40.496884 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.496836 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98k5c\" (UniqueName: \"kubernetes.io/projected/06728ec3-24de-459c-b966-8b88db3bc248-kube-api-access-98k5c\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.496884 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.496837 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh7nl\" (UniqueName: \"kubernetes.io/projected/73f24aac-5672-4b00-b199-324e3da68852-kube-api-access-fh7nl\") pod \"monitoring-plugin-77f4f76594-52pfh\" (UID: \"73f24aac-5672-4b00-b199-324e3da68852\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:40.498334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.498317 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxjr\" (UniqueName: \"kubernetes.io/projected/471a887a-a065-4aa3-9063-555b93a9a258-kube-api-access-kpxjr\") pod \"network-metrics-daemon-psmsw\" (UID: \"471a887a-a065-4aa3-9063-555b93a9a258\") " pod="openshift-multus/network-metrics-daemon-psmsw" Nov 03 19:57:40.500078 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.500058 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnwr\" (UniqueName: \"kubernetes.io/projected/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-kube-api-access-9gnwr\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.502056 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.502034 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"console-b794d6b84-mrbt9\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:40.502457 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.502435 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x8q\" (UniqueName: \"kubernetes.io/projected/426ca1a6-344a-4f33-9e58-50be798ed0a9-kube-api-access-28x8q\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.502497 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.502484 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg24s\" (UniqueName: \"kubernetes.io/projected/2790512d-386a-41e8-8bb6-31ec8faafbc6-kube-api-access-bg24s\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.502538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.502498 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blbkp\" (UniqueName: \"kubernetes.io/projected/fc34dba5-14d2-402c-99ed-0a27e607616d-kube-api-access-blbkp\") pod \"prometheus-operator-9cd89b74c-6vrqs\" (UID: \"fc34dba5-14d2-402c-99ed-0a27e607616d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6vrqs" Nov 03 19:57:40.502606 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.502591 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfwm\" (UniqueName: \"kubernetes.io/projected/47f13c13-8d81-472c-8a39-2f14c8da0242-kube-api-access-4gfwm\") pod \"network-check-target-r97c6\" (UID: \"47f13c13-8d81-472c-8a39-2f14c8da0242\") " pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:40.503778 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.503760 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgr8h\" (UniqueName: \"kubernetes.io/projected/945ef139-c1e8-4154-b869-f18adcc62d8a-kube-api-access-pgr8h\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.503925 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.503909 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"alertmanager-main-0\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:57:40.504924 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.504908 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkqf\" (UniqueName: \"kubernetes.io/projected/31e1076f-26b7-40eb-a11f-b9eca41ee6fc-kube-api-access-9hkqf\") pod \"kube-state-metrics-5dc8d7d89b-5slgw\" (UID: \"31e1076f-26b7-40eb-a11f-b9eca41ee6fc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5slgw" Nov 03 19:57:40.506810 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.506749 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzfzx\" (UniqueName: \"kubernetes.io/projected/3371a194-9a9e-4a3a-9af3-48d86152b1dc-kube-api-access-gzfzx\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.507013 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.506990 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlsx\" (UniqueName: \"kubernetes.io/projected/25f8e8c2-cae1-4e2a-92e0-427d4c4f4186-kube-api-access-rzlsx\") pod \"ingress-canary-8slb6\" (UID: \"25f8e8c2-cae1-4e2a-92e0-427d4c4f4186\") " pod="openshift-ingress-canary/ingress-canary-8slb6" Nov 03 19:57:40.507085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.507020 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcl46\" (UniqueName: \"kubernetes.io/projected/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-kube-api-access-qcl46\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.507150 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.507129 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlc9\" (UniqueName: \"kubernetes.io/projected/32c657c7-cf56-4ca8-b035-fd7c55fc2480-kube-api-access-mjlc9\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.507205 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.507184 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mqs\" (UniqueName: \"kubernetes.io/projected/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kube-api-access-q6mqs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.507316 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.507302 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zkt\" (UniqueName: \"kubernetes.io/projected/78dccc20-8871-4155-aa48-7ffbf940209f-kube-api-access-q5zkt\") pod \"metrics-server-5687c6f496-d89k4\" (UID: \"78dccc20-8871-4155-aa48-7ffbf940209f\") " pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:40.507496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.507480 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"prometheus-k8s-0\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:40.591203 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591170 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591203 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591207 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591228 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591263 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591279 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591289 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-kubelet\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591296 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591321 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-netd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591348 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591349 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-system-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591416 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591373 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-ovn\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591428 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591438 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-slash\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591474 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591505 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591535 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-host\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591540 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591604 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591631 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-netns\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591631 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591657 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591676 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591683 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-kubelet-config\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591712 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591717 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-cnibin\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591720 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-kubernetes\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591733 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-sys-fs\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591746 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591772 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591792 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591811 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591818 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591824 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-bin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.591817 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591831 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591839 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-sys\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591853 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591861 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591863 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-var-lib-kubelet\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591881 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-cni-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591904 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-run-systemd\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591935 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591956 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c60f9fad-b8c0-4d6e-b755-c2bda3f2d014-host\") pod \"node-ca-87bj5\" (UID: \"c60f9fad-b8c0-4d6e-b755-c2bda3f2d014\") " pod="openshift-image-registry/node-ca-87bj5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591964 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.591996 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592022 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592043 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-kubelet\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592047 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592067 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-sys\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592076 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592077 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-var-lib-cni-multus\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592093 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-run\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592129 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592150 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-hostroot\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592169 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592143 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592193 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592227 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592248 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592251 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e97b6a-be41-42af-90ec-bd56e6d791d6-dbus\") pod \"global-pull-secret-syncer-lb594\" (UID: \"18e97b6a-be41-42af-90ec-bd56e6d791d6\") " pod="kube-system/global-pull-secret-syncer-lb594" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592278 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592281 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-etc-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592305 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592310 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysctl-conf\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592314 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3371a194-9a9e-4a3a-9af3-48d86152b1dc-hosts-file\") pod \"node-resolver-v2tvg\" (UID: \"3371a194-9a9e-4a3a-9af3-48d86152b1dc\") " pod="openshift-dns/node-resolver-v2tvg" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592329 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592351 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-socket-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592352 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592382 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-modprobe-d\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592383 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-cnibin\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592384 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592402 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-systemd-units\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592388 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-conf-dir\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592436 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592450 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592487 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592497 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/945ef139-c1e8-4154-b869-f18adcc62d8a-host-slash\") pod \"iptables-alerter-tnxjb\" (UID: \"945ef139-c1e8-4154-b869-f18adcc62d8a\") " pod="openshift-network-operator/iptables-alerter-tnxjb" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592519 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-device-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592532 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592548 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592562 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592593 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592604 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-var-lib-openvswitch\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592608 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592629 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-etc-kubernetes\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592631 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-root\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592634 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592612 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-log-socket\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592644 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-systemd\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592648 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-netns\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592666 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592680 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e458f2d3-a88a-44b2-8bd9-57bbb097bfa8-registration-dir\") pod \"aws-ebs-csi-driver-node-m5v2d\" (UID: \"e458f2d3-a88a-44b2-8bd9-57bbb097bfa8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5v2d" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592732 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-multus-socket-dir-parent\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592738 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592761 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592779 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.592762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592797 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592811 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592814 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-host-cni-bin\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592839 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-system-cni-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592846 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592851 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2790512d-386a-41e8-8bb6-31ec8faafbc6-crio-socket\") pod \"insights-runtime-extractor-ltbwq\" (UID: \"2790512d-386a-41e8-8bb6-31ec8faafbc6\") " pod="openshift-insights/insights-runtime-extractor-ltbwq" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592866 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/06728ec3-24de-459c-b966-8b88db3bc248-node-log\") pod \"ovnkube-node-l9nr5\" (UID: \"06728ec3-24de-459c-b966-8b88db3bc248\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592872 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-etc-sysconfig\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592867 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/426ca1a6-344a-4f33-9e58-50be798ed0a9-node-exporter-wtmp\") pod \"node-exporter-k7sbn\" (UID: \"426ca1a6-344a-4f33-9e58-50be798ed0a9\") " pod="openshift-monitoring/node-exporter-k7sbn" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592873 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592893 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-k8s-cni-cncf-io\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592905 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592920 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592935 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592962 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32c657c7-cf56-4ca8-b035-fd7c55fc2480-host-run-multus-certs\") pod \"multus-ndfvw\" (UID: \"32c657c7-cf56-4ca8-b035-fd7c55fc2480\") " pod="openshift-multus/multus-ndfvw" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592991 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4cbdc4a-e9f3-4489-bd72-341746fdc0c9-lib-modules\") pod \"tuned-49krk\" (UID: \"f4cbdc4a-e9f3-4489-bd72-341746fdc0c9\") " pod="openshift-cluster-node-tuning-operator/tuned-49krk" Nov 03 19:57:40.594475 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:40.592999 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9edf52cd-afe4-46cb-b9b3-148d5445ca6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njwcg\" (UID: \"9edf52cd-afe4-46cb-b9b3-148d5445ca6e\") " pod="openshift-multus/multus-additional-cni-plugins-njwcg" Nov 03 19:57:41.063084 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:41.063045 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:41.166349 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:41.166315 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:41.502759 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:41.502731 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-52pfh" Nov 03 19:57:41.851217 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:41.851194 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:57:42.071064 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:42.071037 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-chgrg" Nov 03 19:57:44.895506 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:44.895473 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:44.911907 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:44.911874 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:46.114309 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.114268 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:57:46.209628 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.209592 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:46.309368 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.309334 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:46.511513 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.511436 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:46.696210 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.696175 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:57:46.709262 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:46.709229 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:47.448051 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:47.448021 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:48.774429 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:48.774396 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:48.872806 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:48.872776 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:49.117595 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:49.117561 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:49.219770 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:49.219740 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:49.783505 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:49.783473 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:57:49.783865 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:49.783709 5954 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:57:52.269855 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:52.269826 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8wq99" Nov 03 19:57:52.691551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:52.691515 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:52.791856 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:52.791825 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:54.691275 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:54.691247 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5687c6f496-d89k4" Nov 03 19:57:56.080298 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:56.080262 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:56.177485 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:57:56.177448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:00.257516 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:00.257479 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 19:58:00.813061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:00.813033 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9nr5" Nov 03 19:58:05.392430 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.392364 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f46b5684f-zmphc" podUID="488637a9-aaf1-4506-a7ad-5d20665bb414" containerName="console" containerID="cri-o://ac1b1f500f9632f9d428cb2ea476327003e4d40ff0bd3ab57d1f54e4156d398e" gracePeriod=15 Nov 03 19:58:05.392872 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.392546 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b85776785-crvbv" podUID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" containerName="console" containerID="cri-o://965d861c14fd1ff5092928f4bb9f86c5b9228c0adab48b05844ab3c531733990" gracePeriod=15 Nov 03 19:58:05.438691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.438659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:05.539635 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.539604 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:05.566799 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.566762 5954 generic.go:356] "Generic (PLEG): container finished" podID="488637a9-aaf1-4506-a7ad-5d20665bb414" containerID="ac1b1f500f9632f9d428cb2ea476327003e4d40ff0bd3ab57d1f54e4156d398e" exitCode=0 Nov 03 19:58:05.566989 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.566839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f46b5684f-zmphc" event={"ID":"488637a9-aaf1-4506-a7ad-5d20665bb414","Type":"ContainerDied","Data":"ac1b1f500f9632f9d428cb2ea476327003e4d40ff0bd3ab57d1f54e4156d398e"} Nov 03 19:58:05.568563 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.568537 5954 generic.go:356] "Generic (PLEG): container finished" podID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" containerID="965d861c14fd1ff5092928f4bb9f86c5b9228c0adab48b05844ab3c531733990" exitCode=0 Nov 03 19:58:05.568708 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.568593 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b85776785-crvbv" event={"ID":"5ea0e226-87c0-40c9-b1b3-6029f76e6576","Type":"ContainerDied","Data":"965d861c14fd1ff5092928f4bb9f86c5b9228c0adab48b05844ab3c531733990"} Nov 03 19:58:05.677781 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.677755 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:58:05.703879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.703846 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:58:05.838192 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838154 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838192 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838199 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838225 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838256 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838284 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838303 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838338 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838365 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838724 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838512 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:05.838724 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838628 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838724 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838698 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838737 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838742 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838778 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838773 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838835 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.838879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838865 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.839196 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838893 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") pod \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\" (UID: \"5ea0e226-87c0-40c9-b1b3-6029f76e6576\") " Nov 03 19:58:05.839196 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.838921 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") pod \"488637a9-aaf1-4506-a7ad-5d20665bb414\" (UID: \"488637a9-aaf1-4506-a7ad-5d20665bb414\") " Nov 03 19:58:05.839196 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839149 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839188 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839197 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839252 5954 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488637a9-aaf1-4506-a7ad-5d20665bb414-tmp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839273 5954 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839291 5954 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-oauth-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839307 5954 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-trusted-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.839358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839321 5954 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ea0e226-87c0-40c9-b1b3-6029f76e6576-tmp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.839708 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839517 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.839708 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.839572 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:05.840941 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.840915 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "kube-api-access-q554h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:05.841278 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.841253 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:05.841376 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.841352 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:05.841442 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.841389 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:05.841500 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.841465 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj" pod "488637a9-aaf1-4506-a7ad-5d20665bb414" (UID: "488637a9-aaf1-4506-a7ad-5d20665bb414"). InnerVolumeSpecName "kube-api-access-7h9jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:05.841590 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.841572 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert" pod "5ea0e226-87c0-40c9-b1b3-6029f76e6576" (UID: "5ea0e226-87c0-40c9-b1b3-6029f76e6576"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940453 5954 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-oauth-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940487 5954 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940500 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h9jj\" (UniqueName: \"kubernetes.io/projected/488637a9-aaf1-4506-a7ad-5d20665bb414-kube-api-access-7h9jj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940513 5954 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-oauth-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940525 5954 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940539 5954 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ea0e226-87c0-40c9-b1b3-6029f76e6576-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940551 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940552 5954 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/488637a9-aaf1-4506-a7ad-5d20665bb414-console-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940895 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940565 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q554h\" (UniqueName: \"kubernetes.io/projected/5ea0e226-87c0-40c9-b1b3-6029f76e6576-kube-api-access-q554h\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940895 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940577 5954 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ea0e226-87c0-40c9-b1b3-6029f76e6576-console-oauth-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:05.940895 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:05.940590 5954 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/488637a9-aaf1-4506-a7ad-5d20665bb414-console-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:06.022768 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.022737 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:06.573518 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.573485 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f46b5684f-zmphc" Nov 03 19:58:06.573518 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.573506 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f46b5684f-zmphc" event={"ID":"488637a9-aaf1-4506-a7ad-5d20665bb414","Type":"ContainerDied","Data":"3291015c61d2ddc80ce4082db8bdd80796c74ebdfa59a6eed595e3ebf24041c9"} Nov 03 19:58:06.574025 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.573581 5954 scope.go:122] "RemoveContainer" containerID="ac1b1f500f9632f9d428cb2ea476327003e4d40ff0bd3ab57d1f54e4156d398e" Nov 03 19:58:06.575035 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.574856 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b85776785-crvbv" event={"ID":"5ea0e226-87c0-40c9-b1b3-6029f76e6576","Type":"ContainerDied","Data":"57784f1d0095db58ea77fc9e25116764fe94abd271922b18cfba15c3a48aa40c"} Nov 03 19:58:06.575035 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.574885 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b85776785-crvbv" Nov 03 19:58:06.593092 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.593070 5954 scope.go:122] "RemoveContainer" containerID="965d861c14fd1ff5092928f4bb9f86c5b9228c0adab48b05844ab3c531733990" Nov 03 19:58:06.598555 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.598528 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:58:06.604690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.604664 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f46b5684f-zmphc"] Nov 03 19:58:06.617292 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.617263 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:58:06.621423 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:06.621397 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b85776785-crvbv"] Nov 03 19:58:07.401509 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:07.401481 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488637a9-aaf1-4506-a7ad-5d20665bb414" path="/var/lib/kubelet/pods/488637a9-aaf1-4506-a7ad-5d20665bb414/volumes" Nov 03 19:58:07.401888 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:07.401875 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" path="/var/lib/kubelet/pods/5ea0e226-87c0-40c9-b1b3-6029f76e6576/volumes" Nov 03 19:58:08.688885 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:08.688852 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:08.790345 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:08.790312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:11.065154 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:11.065095 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:11.165740 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:11.165707 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:16.699392 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:16.699360 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r97c6" Nov 03 19:58:30.257791 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:30.257750 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 19:58:46.130261 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:46.130231 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:46.559995 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:46.559913 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:50.441010 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:50.440975 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:50.663843 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:50.663806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:51.525900 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.525867 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526192 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="alertmanager" containerID="cri-o://f1caaa4597cd1d72fb8bba5c95b4de86e587a36dd42bdb9400b65c478d62bcd8" gracePeriod=120 Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526203 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-metric" containerID="cri-o://3cd18385c076dc354a28253a87a6afad04c3437655dbd2161c8aa2ea5ef07d05" gracePeriod=120 Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526281 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="config-reloader" containerID="cri-o://eec2ed1542d04fd24ffddc22553e3dd47bc7d254604dad49989e148a6853e9b8" gracePeriod=120 Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526202 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-web" containerID="cri-o://18ab646cae9c0d881559e7ba04f7f42edb2e03d81befcc89a85639d0188a3d24" gracePeriod=120 Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526377 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="prom-label-proxy" containerID="cri-o://35718d8b301a82b15b3650cb427837a2c611b697b68c0ea4d80f82c3bdfd8cde" gracePeriod=120 Nov 03 19:58:51.526453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.526243 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy" containerID="cri-o://b6fa5b8aa2b08aff66e9a3fa686105c0c5c6d6db5dd905248769fdecea990e1e" gracePeriod=120 Nov 03 19:58:51.756249 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756218 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="35718d8b301a82b15b3650cb427837a2c611b697b68c0ea4d80f82c3bdfd8cde" exitCode=0 Nov 03 19:58:51.756249 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756244 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="3cd18385c076dc354a28253a87a6afad04c3437655dbd2161c8aa2ea5ef07d05" exitCode=0 Nov 03 19:58:51.756249 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756252 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="b6fa5b8aa2b08aff66e9a3fa686105c0c5c6d6db5dd905248769fdecea990e1e" exitCode=0 Nov 03 19:58:51.756249 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756261 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="eec2ed1542d04fd24ffddc22553e3dd47bc7d254604dad49989e148a6853e9b8" exitCode=0 Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756268 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="f1caaa4597cd1d72fb8bba5c95b4de86e587a36dd42bdb9400b65c478d62bcd8" exitCode=0 Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756297 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"35718d8b301a82b15b3650cb427837a2c611b697b68c0ea4d80f82c3bdfd8cde"} Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756374 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"3cd18385c076dc354a28253a87a6afad04c3437655dbd2161c8aa2ea5ef07d05"} Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756401 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"b6fa5b8aa2b08aff66e9a3fa686105c0c5c6d6db5dd905248769fdecea990e1e"} Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756432 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"eec2ed1542d04fd24ffddc22553e3dd47bc7d254604dad49989e148a6853e9b8"} Nov 03 19:58:51.756496 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:51.756457 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"f1caaa4597cd1d72fb8bba5c95b4de86e587a36dd42bdb9400b65c478d62bcd8"} Nov 03 19:58:52.763428 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.763397 5954 generic.go:356] "Generic (PLEG): container finished" podID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerID="18ab646cae9c0d881559e7ba04f7f42edb2e03d81befcc89a85639d0188a3d24" exitCode=0 Nov 03 19:58:52.763770 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.763436 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"18ab646cae9c0d881559e7ba04f7f42edb2e03d81befcc89a85639d0188a3d24"} Nov 03 19:58:52.763770 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.763536 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0adf7348-9f76-4a5c-889a-4f687fe056b2","Type":"ContainerDied","Data":"559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae"} Nov 03 19:58:52.763770 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.763579 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559dbacc8f6f9d18db1cf3c8a641448262d836727c30b8d05b8e1cf830c967ae" Nov 03 19:58:52.767695 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.767676 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:52.829818 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829788 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.829991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829832 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.829991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829870 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.829991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829900 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.829991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829934 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.829991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.829982 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830298 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830013 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830359 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830306 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:52.830552 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830502 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:52.830647 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830617 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830698 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830682 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830747 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830723 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830847 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830829 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830921 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830904 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.830973 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.830952 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"0adf7348-9f76-4a5c-889a-4f687fe056b2\" (UID: \"0adf7348-9f76-4a5c-889a-4f687fe056b2\") " Nov 03 19:58:52.831257 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.831212 5954 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-main-db\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.831257 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.831236 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-metrics-client-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.832072 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.832048 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:52.833033 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.832998 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:52.833725 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.833690 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.833831 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.833798 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.834066 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.834040 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "kube-api-access-846qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:52.834936 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.834910 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:52.835235 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.835215 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.835529 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.835507 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.836930 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.836909 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.839733 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.839702 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.845806 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.845773 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config" pod "0adf7348-9f76-4a5c-889a-4f687fe056b2" (UID: "0adf7348-9f76-4a5c-889a-4f687fe056b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932050 5954 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-cluster-tls-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932089 5954 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-main-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932103 5954 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932142 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-846qk\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-kube-api-access-846qk\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932156 5954 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932173 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932171 5954 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0adf7348-9f76-4a5c-889a-4f687fe056b2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932479 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932186 5954 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932479 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932200 5954 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932479 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932214 5954 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0adf7348-9f76-4a5c-889a-4f687fe056b2-web-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932479 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932225 5954 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0adf7348-9f76-4a5c-889a-4f687fe056b2-config-out\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:52.932479 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:52.932236 5954 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0adf7348-9f76-4a5c-889a-4f687fe056b2-tls-assets\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:53.767280 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.767187 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.788326 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.788298 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:53.793989 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.793962 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:53.823188 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823157 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:53.823515 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823498 5954 cpu_manager.go:414] "containerMap: removing container" podUID="488637a9-aaf1-4506-a7ad-5d20665bb414" containerName="console" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823517 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488637a9-aaf1-4506-a7ad-5d20665bb414" containerName="console" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823535 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="prom-label-proxy" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823542 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="prom-label-proxy" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823552 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="config-reloader" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823559 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="config-reloader" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823574 5954 cpu_manager.go:414] "containerMap: removing container" podUID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" containerName="console" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823582 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" containerName="console" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823595 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="init-config-reloader" Nov 03 19:58:53.823604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823602 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="init-config-reloader" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823633 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="alertmanager" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823640 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="alertmanager" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823653 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-metric" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823661 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-metric" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823671 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823678 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823697 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-web" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823704 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-web" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823779 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-web" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823791 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="488637a9-aaf1-4506-a7ad-5d20665bb414" containerName="console" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823803 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="prom-label-proxy" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823812 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="config-reloader" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823821 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea0e226-87c0-40c9-b1b3-6029f76e6576" containerName="console" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823830 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="init-config-reloader" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823841 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="alertmanager" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823849 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy-metric" Nov 03 19:58:53.824085 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.823857 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" containerName="kube-rbac-proxy" Nov 03 19:58:53.827403 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.827382 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.831354 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831334 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 19:58:53.831484 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831337 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831355 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831422 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831441 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831512 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831512 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 19:58:53.831581 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831514 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 19:58:53.831956 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.831940 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wbnrm\"" Nov 03 19:58:53.835075 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.835058 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 19:58:53.847038 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.847018 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:53.942880 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.942851 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943070 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.942890 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943070 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.942907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943070 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.942925 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fknw\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-kube-api-access-2fknw\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943070 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943050 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-web-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943307 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943084 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943307 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943132 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943307 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943185 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943440 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943311 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943440 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943334 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-config-out\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943440 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943389 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-config-volume\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943440 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943431 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:53.943592 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:53.943463 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044705 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044637 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-config-out\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044705 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044680 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-config-volume\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044705 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044703 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044725 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044766 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044804 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044834 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044855 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fknw\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-kube-api-access-2fknw\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.044903 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044882 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-web-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044916 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044950 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.044975 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.045019 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.045078 5954 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:58:54.045227 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.045203 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.046762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.045644 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.046762 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.046605 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44071c1c-bee7-426f-820d-436d709abcc2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048028 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.047872 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44071c1c-bee7-426f-820d-436d709abcc2-config-out\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048028 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048021 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048225 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048022 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-web-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048288 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048260 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048348 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048291 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048396 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048353 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-config-volume\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.048680 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.048659 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.049126 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.049088 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.049744 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.049719 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44071c1c-bee7-426f-820d-436d709abcc2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.057939 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.057914 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fknw\" (UniqueName: \"kubernetes.io/projected/44071c1c-bee7-426f-820d-436d709abcc2-kube-api-access-2fknw\") pod \"alertmanager-main-0\" (UID: \"44071c1c-bee7-426f-820d-436d709abcc2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.145469 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.145426 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 19:58:54.429753 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.429719 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 19:58:54.433433 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:58:54.433401 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44071c1c_bee7_426f_820d_436d709abcc2.slice/crio-2fc91ea917ab0f94b3c8640e92eee55118cafcaaa2b0ea2ea3f6c4488cecf800 WatchSource:0}: Error finding container 2fc91ea917ab0f94b3c8640e92eee55118cafcaaa2b0ea2ea3f6c4488cecf800: Status 404 returned error can't find the container with id 2fc91ea917ab0f94b3c8640e92eee55118cafcaaa2b0ea2ea3f6c4488cecf800 Nov 03 19:58:54.772027 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.771938 5954 generic.go:356] "Generic (PLEG): container finished" podID="44071c1c-bee7-426f-820d-436d709abcc2" containerID="0d2d1c97c8bad9eaf99322ba9e4ca3409de61d1d0d703f0a16fd1188b26ceefa" exitCode=0 Nov 03 19:58:54.772027 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.772000 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerDied","Data":"0d2d1c97c8bad9eaf99322ba9e4ca3409de61d1d0d703f0a16fd1188b26ceefa"} Nov 03 19:58:54.772437 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:54.772052 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"2fc91ea917ab0f94b3c8640e92eee55118cafcaaa2b0ea2ea3f6c4488cecf800"} Nov 03 19:58:55.402069 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.402034 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adf7348-9f76-4a5c-889a-4f687fe056b2" path="/var/lib/kubelet/pods/0adf7348-9f76-4a5c-889a-4f687fe056b2/volumes" Nov 03 19:58:55.541604 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.541570 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dbb5848c7-wcspz"] Nov 03 19:58:55.544231 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.544199 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.548605 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.548579 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 19:58:55.552830 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.552803 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 19:58:55.561686 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.561532 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dbb5848c7-wcspz"] Nov 03 19:58:55.611225 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.611195 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 19:58:55.611545 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.611236 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 19:58:55.611545 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.611237 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 19:58:55.611545 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.611268 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vm77v\"" Nov 03 19:58:55.611545 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.611292 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 19:58:55.661968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.661880 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-metrics-client-ca\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.661968 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.661934 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-serving-certs-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662177 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662046 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662177 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662102 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662177 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662164 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-federate-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662282 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662180 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662317 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662287 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46kl\" (UniqueName: \"kubernetes.io/projected/b769fedf-c73d-4aad-90dc-a8a49d0693dc-kube-api-access-w46kl\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.662357 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.662333 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763145 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763092 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46kl\" (UniqueName: \"kubernetes.io/projected/b769fedf-c73d-4aad-90dc-a8a49d0693dc-kube-api-access-w46kl\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763330 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763165 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763330 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763195 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-metrics-client-ca\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763330 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763213 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-serving-certs-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763330 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763237 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763505 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763372 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763505 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763491 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-federate-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.763606 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.763524 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.764061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.764008 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-serving-certs-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.764299 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.764276 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-metrics-client-ca\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.764583 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.764560 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.765830 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.765812 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.766022 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.765996 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-federate-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.766168 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.766153 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-telemeter-client-tls\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.766471 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.766454 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b769fedf-c73d-4aad-90dc-a8a49d0693dc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.775258 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.775233 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46kl\" (UniqueName: \"kubernetes.io/projected/b769fedf-c73d-4aad-90dc-a8a49d0693dc-kube-api-access-w46kl\") pod \"telemeter-client-5dbb5848c7-wcspz\" (UID: \"b769fedf-c73d-4aad-90dc-a8a49d0693dc\") " pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:55.779382 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779359 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"9c10fae3cdba7b0f7bd6c887f81647b87c4a340caaa580bc3fdfd2c74bb76a54"} Nov 03 19:58:55.779490 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779415 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"d06154947e6f0c34abbcd87111b3d419843b37b4e633868290f401096ee3b192"} Nov 03 19:58:55.779490 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779439 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"2f537a58bff20d731ef47fabbfb254a7a7f8f82de274659bc0c8058cd842f69d"} Nov 03 19:58:55.779490 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779464 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"6d122f467f042337b4950b12cba6b851c1617602a31456b2d4498a456c900a04"} Nov 03 19:58:55.779490 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779486 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"2edd68b8ab47e6f0e107e992f9d7ef0921f1fc58b7b63876223d906ce861a01e"} Nov 03 19:58:55.779702 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.779509 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44071c1c-bee7-426f-820d-436d709abcc2","Type":"ContainerStarted","Data":"e2d66bc357b60424cdb386bbf22130b7cb725019666f38e954c2fc3160533c65"} Nov 03 19:58:55.811993 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.811937 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.800085533 podStartE2EDuration="2.811918945s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.833412ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:55.811385415 +0000 UTC m=+76.536393045" watchObservedRunningTime="2026-11-03 19:58:55.811918945 +0000 UTC m=+76.536926542" Nov 03 19:58:55.841315 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.841278 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:55.841671 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.841645 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="prometheus" containerID="cri-o://69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" gracePeriod=600 Nov 03 19:58:55.842062 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.842034 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-thanos" containerID="cri-o://48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" gracePeriod=600 Nov 03 19:58:55.842185 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.842168 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy" containerID="cri-o://df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" gracePeriod=600 Nov 03 19:58:55.842601 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.842246 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-web" containerID="cri-o://5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" gracePeriod=600 Nov 03 19:58:55.842601 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.842315 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="thanos-sidecar" containerID="cri-o://80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" gracePeriod=600 Nov 03 19:58:55.842601 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.842385 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="config-reloader" containerID="cri-o://3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" gracePeriod=600 Nov 03 19:58:55.860161 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:55.860129 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" Nov 03 19:58:56.095031 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.095003 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.156599 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.156564 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dbb5848c7-wcspz"] Nov 03 19:58:56.161364 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:58:56.161333 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb769fedf_c73d_4aad_90dc_a8a49d0693dc.slice/crio-c42b4ccb0d693e2a5ff948ad0c910da2c53ca368831357ffec44d1c5bb27b57d WatchSource:0}: Error finding container c42b4ccb0d693e2a5ff948ad0c910da2c53ca368831357ffec44d1c5bb27b57d: Status 404 returned error can't find the container with id c42b4ccb0d693e2a5ff948ad0c910da2c53ca368831357ffec44d1c5bb27b57d Nov 03 19:58:56.163403 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.163386 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:58:56.167738 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167659 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167738 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167706 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167738 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167731 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167772 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167792 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167832 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167861 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.167894 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167893 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167911 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167933 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167969 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.167991 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.168018 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.168061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.168037 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169080 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.168762 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.169198 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169177 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169258 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169240 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169275 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169307 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169351 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169517 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169385 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") pod \"ad270548-2169-4822-8c43-17f547dc0421\" (UID: \"ad270548-2169-4822-8c43-17f547dc0421\") " Nov 03 19:58:56.169704 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.169677 5954 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-metrics-client-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.170405 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.170374 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.170489 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.170471 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.172142 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172029 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.172142 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172071 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "kube-api-access-jspk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:56.172142 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172088 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.172337 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172251 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:56.172337 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172259 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.172337 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172334 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.172595 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172574 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.172768 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.172746 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.173225 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.173200 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.173405 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.173385 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.173801 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.173777 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:58:56.173887 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.173778 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.174895 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.174854 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.175035 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.175012 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:58:56.176500 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.176269 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:58:56.177783 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.177759 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.190015 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.189454 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config" pod "ad270548-2169-4822-8c43-17f547dc0421" (UID: "ad270548-2169-4822-8c43-17f547dc0421"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:58:56.271769 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271729 5954 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271769 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271759 5954 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-metrics-client-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271769 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271770 5954 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271782 5954 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271792 5954 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271801 5954 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-tls-assets\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271810 5954 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-config-out\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271818 5954 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271827 5954 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271835 5954 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-db\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271844 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspk2\" (UniqueName: \"kubernetes.io/projected/ad270548-2169-4822-8c43-17f547dc0421-kube-api-access-jspk2\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271852 5954 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271862 5954 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-kube-rbac-proxy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271870 5954 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271880 5954 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271888 5954 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-web-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271919 5954 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad270548-2169-4822-8c43-17f547dc0421-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271933 5954 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-secret-grpc-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.271988 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.271946 5954 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad270548-2169-4822-8c43-17f547dc0421-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:58:56.790985 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.790951 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" exitCode=0 Nov 03 19:58:56.790985 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.790981 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" exitCode=0 Nov 03 19:58:56.790985 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.790990 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" exitCode=0 Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.790998 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" exitCode=0 Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791005 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" exitCode=0 Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791012 5954 generic.go:356] "Generic (PLEG): container finished" podID="ad270548-2169-4822-8c43-17f547dc0421" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" exitCode=0 Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791039 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791035 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791151 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791165 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791186 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791223 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791259 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791284 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} Nov 03 19:58:56.791538 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.791310 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad270548-2169-4822-8c43-17f547dc0421","Type":"ContainerDied","Data":"78ecdd1176e8349027c1fff1b6368c7b0ddb0ea5fc6a0a5ff5a59769345f0ad7"} Nov 03 19:58:56.793360 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.793338 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" event={"ID":"b769fedf-c73d-4aad-90dc-a8a49d0693dc","Type":"ContainerStarted","Data":"c42b4ccb0d693e2a5ff948ad0c910da2c53ca368831357ffec44d1c5bb27b57d"} Nov 03 19:58:56.800255 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.800242 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.807646 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.807628 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.814993 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.814968 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.817249 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.817228 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:56.822025 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.821996 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:56.824030 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.824011 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.830981 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.830964 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.838071 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.838052 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.847295 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.847277 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.847850 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.847817 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.847927 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.847857 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.847927 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.847911 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.848236 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.848212 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.848321 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.848245 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.848321 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.848267 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.848978 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.848952 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.849058 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.848982 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.849058 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.849002 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.849338 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.849318 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.849400 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.849344 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.849400 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.849362 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.849757 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.849735 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.849853 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.849760 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.849853 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.849777 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.850077 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.850054 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.850169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850086 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.850169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850128 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.850448 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:58:56.850431 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.850494 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850452 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.850494 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850466 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.850744 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850724 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.850744 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850743 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.851016 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.850990 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.851074 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851017 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.851268 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851243 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.851324 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851270 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.851618 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851586 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.851618 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851608 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.851872 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851848 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.851941 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.851874 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.852159 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852095 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.852237 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852160 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.852451 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852431 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.852534 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852452 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.852735 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852708 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.852735 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852726 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.852954 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852938 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.853014 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.852954 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.853235 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853207 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.853235 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853232 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.853470 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853454 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.853470 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853470 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.853711 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853692 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.853785 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.853711 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.854059 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854040 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.854162 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854060 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.854371 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854347 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.854445 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854373 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.854618 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854599 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.854673 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854619 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.854919 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854860 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.854919 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.854884 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.855149 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.855102 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.855149 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.855148 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.856189 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.855753 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.856189 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.855802 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.856352 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856296 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.856352 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856321 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.856599 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856579 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.856678 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856600 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.856734 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856683 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:56.856845 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856826 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.856922 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.856847 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.857052 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857034 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="prometheus" Nov 03 19:58:56.857174 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857054 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="prometheus" Nov 03 19:58:56.857174 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857072 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-thanos" Nov 03 19:58:56.857174 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857081 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-thanos" Nov 03 19:58:56.857174 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857121 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.857174 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857142 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857129 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="init-config-reloader" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857199 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="init-config-reloader" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857225 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-web" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857234 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-web" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857244 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="config-reloader" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857251 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="config-reloader" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857273 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="thanos-sidecar" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857280 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="thanos-sidecar" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857298 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857305 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857389 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="thanos-sidecar" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857403 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857412 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="prometheus" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857421 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-thanos" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857432 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="init-config-reloader" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857444 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="kube-rbac-proxy-web" Nov 03 19:58:56.857463 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.857453 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad270548-2169-4822-8c43-17f547dc0421" containerName="config-reloader" Nov 03 19:58:56.859450 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859427 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.859450 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859451 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.859683 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859664 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.859745 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859682 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.859945 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859921 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.860014 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.859945 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.860360 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860338 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.860432 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860360 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.860610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860589 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.860677 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860611 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.860821 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860796 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.860821 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.860820 5954 scope.go:122] "RemoveContainer" containerID="48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb" Nov 03 19:58:56.861233 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861210 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb"} err="failed to get container status \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": rpc error: code = NotFound desc = could not find container \"48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb\": container with ID starting with 48802a51f8daf38212dffdf545b4cb0214a6249807103db1129e8912a36824cb not found: ID does not exist" Nov 03 19:58:56.861312 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861233 5954 scope.go:122] "RemoveContainer" containerID="df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e" Nov 03 19:58:56.861444 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861422 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e"} err="failed to get container status \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": rpc error: code = NotFound desc = could not find container \"df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e\": container with ID starting with df5b6700cfa3da6fbe548cba5dd0c36b71fb3ac2ecb82e5432fffc9f4ca8d86e not found: ID does not exist" Nov 03 19:58:56.861499 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861444 5954 scope.go:122] "RemoveContainer" containerID="5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8" Nov 03 19:58:56.861785 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861755 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.861879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861746 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8"} err="failed to get container status \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": rpc error: code = NotFound desc = could not find container \"5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8\": container with ID starting with 5754ab795442c7f02b4f3afc5c9397653b770ae684381b25633826feb5ad11f8 not found: ID does not exist" Nov 03 19:58:56.861879 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.861854 5954 scope.go:122] "RemoveContainer" containerID="80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8" Nov 03 19:58:56.862180 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.862159 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8"} err="failed to get container status \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": rpc error: code = NotFound desc = could not find container \"80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8\": container with ID starting with 80533d410449bedc2e4e241e2d87837485bc3a2b2a6705120c4a97792c1962c8 not found: ID does not exist" Nov 03 19:58:56.862258 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.862181 5954 scope.go:122] "RemoveContainer" containerID="3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581" Nov 03 19:58:56.862857 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.862825 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581"} err="failed to get container status \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": rpc error: code = NotFound desc = could not find container \"3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581\": container with ID starting with 3e69667df20709442faf34a7afa58cce1a65c74ea90199b3b7f3c086bac54581 not found: ID does not exist" Nov 03 19:58:56.862857 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.862853 5954 scope.go:122] "RemoveContainer" containerID="69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f" Nov 03 19:58:56.863429 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.863394 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f"} err="failed to get container status \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": rpc error: code = NotFound desc = could not find container \"69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f\": container with ID starting with 69919a4ff64ae6275a1a5a9a4e4a1e9b0ede375ac40ab07e8805f4230e11354f not found: ID does not exist" Nov 03 19:58:56.863429 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.863418 5954 scope.go:122] "RemoveContainer" containerID="98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f" Nov 03 19:58:56.863793 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.863751 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f"} err="failed to get container status \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": rpc error: code = NotFound desc = could not find container \"98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f\": container with ID starting with 98f9dce0cfa0ade61a7ea8dcbf478359cfea46ebdd51cab2250dc84dcdddd56f not found: ID does not exist" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.866496 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.866558 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.866577 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.866618 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.866700 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867346 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjsd2\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867357 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 19:58:56.867361 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867361 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 19:58:56.867779 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867376 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 19:58:56.867779 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867401 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 19:58:56.867779 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.867517 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 19:58:56.868955 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.868565 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 19:58:56.868955 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.868620 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 19:58:56.868955 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.868805 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-648mvoststcbk\"" Nov 03 19:58:56.873532 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.873487 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 19:58:56.873630 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.873598 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 19:58:56.880498 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.880420 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:56.978602 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978557 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.978752 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978621 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.978752 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978705 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.978752 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978735 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.978920 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978759 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.978920 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978888 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979019 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.978953 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979077 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979027 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979077 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979063 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979177 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979159 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979224 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979188 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979224 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979204 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979224 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979221 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979277 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979358 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979326 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979444 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979405 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979503 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979472 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979554 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979554 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979541 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:56.979644 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:56.979571 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9xj6\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-kube-api-access-f9xj6\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080422 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080453 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080454 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080476 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080494 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080593 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080627 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080644 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080690 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080667 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080696 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080721 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080757 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080779 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080805 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080856 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080891 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9xj6\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-kube-api-access-f9xj6\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.080991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.080977 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081016 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081031 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081189 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081221 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081249 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.081412 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081352 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.082623 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.081963 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.082623 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.082558 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084398 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.083924 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084398 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084084 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084398 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084100 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084398 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084176 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084398 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084357 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084740 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084616 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.084740 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.084709 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.085819 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086137 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.086086 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086302 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.086272 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c37eff7-3832-4dc4-a825-9d7533ca0db4-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086480 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.086457 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086544 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.086466 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c37eff7-3832-4dc4-a825-9d7533ca0db4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.086904 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.086885 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.087580 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.087555 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.088190 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.088170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c37eff7-3832-4dc4-a825-9d7533ca0db4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.094796 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.094776 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9xj6\" (UniqueName: \"kubernetes.io/projected/0c37eff7-3832-4dc4-a825-9d7533ca0db4-kube-api-access-f9xj6\") pod \"prometheus-k8s-0\" (UID: \"0c37eff7-3832-4dc4-a825-9d7533ca0db4\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.184992 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.184911 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:58:57.403217 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.403185 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad270548-2169-4822-8c43-17f547dc0421" path="/var/lib/kubelet/pods/ad270548-2169-4822-8c43-17f547dc0421/volumes" Nov 03 19:58:57.477658 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.477504 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 19:58:57.482619 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:58:57.482594 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c37eff7_3832_4dc4_a825_9d7533ca0db4.slice/crio-a82bc1d73278210ff33c8bd130b677f64981dce65474b55dd521ce4a9b5ebf01 WatchSource:0}: Error finding container a82bc1d73278210ff33c8bd130b677f64981dce65474b55dd521ce4a9b5ebf01: Status 404 returned error can't find the container with id a82bc1d73278210ff33c8bd130b677f64981dce65474b55dd521ce4a9b5ebf01 Nov 03 19:58:57.798991 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.798901 5954 generic.go:356] "Generic (PLEG): container finished" podID="0c37eff7-3832-4dc4-a825-9d7533ca0db4" containerID="ae7fa5b389cef9c340faf0af9d04e66ee90a4e0b602139ba3e9d6c2424532f35" exitCode=0 Nov 03 19:58:57.799413 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.798992 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerDied","Data":"ae7fa5b389cef9c340faf0af9d04e66ee90a4e0b602139ba3e9d6c2424532f35"} Nov 03 19:58:57.799413 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:57.799064 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"a82bc1d73278210ff33c8bd130b677f64981dce65474b55dd521ce4a9b5ebf01"} Nov 03 19:58:58.807285 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.807244 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" event={"ID":"b769fedf-c73d-4aad-90dc-a8a49d0693dc","Type":"ContainerStarted","Data":"4da88d0e531ed5aa654e50f2afbe74d7711292ce50053601a613889a4ef818ff"} Nov 03 19:58:58.807713 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.807309 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" event={"ID":"b769fedf-c73d-4aad-90dc-a8a49d0693dc","Type":"ContainerStarted","Data":"23f6f2b8cc902fccdb23b00f25e0b67d0ae45124e0d97dac99a776323381d94d"} Nov 03 19:58:58.807713 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.807336 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" event={"ID":"b769fedf-c73d-4aad-90dc-a8a49d0693dc","Type":"ContainerStarted","Data":"898b4c25b6bcb5192006b0179aeaf494419bdf7e5a988a07c6038b77ce0a398b"} Nov 03 19:58:58.810457 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810430 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"7b94cb428a8351a6f6a1bc932d6ed26491c8e53880cf789ded86aebe4a03db21"} Nov 03 19:58:58.810567 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810492 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"5f48dae68c29686763005ce8617a551bbfee54de97a136ef960b8bed7b8be3c3"} Nov 03 19:58:58.810567 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810518 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"f808dfff17ad07a1e009b89064c007853eea22af46b7247109024ce5e0334fd4"} Nov 03 19:58:58.810567 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810542 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"25b5586f6b1b5681b9e70c5712760a8b24e48964bcab4e9e579f3a9e7cc6b1f3"} Nov 03 19:58:58.810567 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810566 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"2ac080fda4b8cec93acad383764fb85f1e78f94d8cd00b0eb64033cd7ba20a35"} Nov 03 19:58:58.810724 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.810591 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c37eff7-3832-4dc4-a825-9d7533ca0db4","Type":"ContainerStarted","Data":"63f36901297b954b7a25d5ce60fae6c8c32919274e2b2b7dddd04ce99274dbf4"} Nov 03 19:58:58.835186 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.835134 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dbb5848c7-wcspz" podStartSLOduration=1.814568532 podStartE2EDuration="3.835097927s" totalImagesPullingTime="2.020529395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:58.833465775 +0000 UTC m=+79.558473372" watchObservedRunningTime="2026-11-03 19:58:58.835097927 +0000 UTC m=+79.560105524" Nov 03 19:58:58.868468 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.868417 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.844053441 podStartE2EDuration="2.868404182s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.350741ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:58.867379368 +0000 UTC m=+79.592386963" watchObservedRunningTime="2026-11-03 19:58:58.868404182 +0000 UTC m=+79.593411778" Nov 03 19:58:58.924127 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.924081 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 19:58:58.926312 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.926292 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:58.938945 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:58.938920 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 19:58:59.015764 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.015732 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:59.103951 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.103924 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104066 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.103965 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104066 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.103985 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104172 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.104126 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104218 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.104167 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104218 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.104192 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jzf\" (UniqueName: \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104276 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.104260 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.104306 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.104282 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.111625 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.111601 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:59.205435 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205406 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205435 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205443 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205478 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205507 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205522 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jzf\" (UniqueName: \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205559 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205580 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.205691 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.205618 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.206171 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.206098 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.206428 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.206405 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.206511 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.206475 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.206547 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.206528 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.206610 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.206593 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.208129 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.208080 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.208242 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.208227 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.223399 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.223374 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jzf\" (UniqueName: \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\") pod \"console-b44ff6d88-q725z\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.237354 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.237338 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:58:59.510996 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.510925 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 19:58:59.514035 ip-10-0-135-70 kubenswrapper[5954]: W1103 19:58:59.514011 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec41c4c_0dd0_4335_bab0_d603af234817.slice/crio-6bc4993fa4c60967b46880d318376c8a3170d4bb25ff28f447389f85fbfb47d2 WatchSource:0}: Error finding container 6bc4993fa4c60967b46880d318376c8a3170d4bb25ff28f447389f85fbfb47d2: Status 404 returned error can't find the container with id 6bc4993fa4c60967b46880d318376c8a3170d4bb25ff28f447389f85fbfb47d2 Nov 03 19:58:59.609170 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.609066 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:59.709037 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.709003 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:59.817000 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.816637 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b44ff6d88-q725z" event={"ID":"dec41c4c-0dd0-4335-bab0-d603af234817","Type":"ContainerStarted","Data":"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b"} Nov 03 19:58:59.817000 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.816695 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b44ff6d88-q725z" event={"ID":"dec41c4c-0dd0-4335-bab0-d603af234817","Type":"ContainerStarted","Data":"6bc4993fa4c60967b46880d318376c8a3170d4bb25ff28f447389f85fbfb47d2"} Nov 03 19:58:59.838180 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:58:59.837422 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b44ff6d88-q725z" podStartSLOduration=1.837408469 podStartE2EDuration="1.837408469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:59.837214541 +0000 UTC m=+80.562222136" watchObservedRunningTime="2026-11-03 19:58:59.837408469 +0000 UTC m=+80.562416062" Nov 03 19:59:00.256317 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:59:00.256281 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 19:59:02.185606 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:02.185576 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:59:09.237465 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:09.237425 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:59:09.237854 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:09.237519 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:59:09.242443 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:09.242421 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:59:09.857620 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:09.857590 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 19:59:09.911888 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:09.911860 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:59:30.256586 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:59:30.256502 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 19:59:34.398126 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:34.398086 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:59:34.495404 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:34.495365 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:59:34.931771 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:34.931734 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b794d6b84-mrbt9" podUID="a0ac0444-b127-4046-80b2-49f8d1a416e5" containerName="console" containerID="cri-o://b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb" gracePeriod=15 Nov 03 19:59:35.174892 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.174867 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:59:35.315259 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315174 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315259 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315224 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315506 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315267 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315506 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315301 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315620 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315515 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315620 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315569 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315620 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315597 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315777 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315629 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") pod \"a0ac0444-b127-4046-80b2-49f8d1a416e5\" (UID: \"a0ac0444-b127-4046-80b2-49f8d1a416e5\") " Nov 03 19:59:35.315777 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315736 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:35.315777 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315740 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:35.315954 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315933 5954 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.316005 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.315960 5954 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-oauth-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.316061 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.316025 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:35.316243 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.316139 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:59:35.316334 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.316255 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:59:35.317633 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.317605 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "kube-api-access-qfnh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:59:35.317633 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.317619 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:59:35.317766 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.317655 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config" pod "a0ac0444-b127-4046-80b2-49f8d1a416e5" (UID: "a0ac0444-b127-4046-80b2-49f8d1a416e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:59:35.416451 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416428 5954 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-oauth-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.416451 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416451 5954 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-trusted-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.416795 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416462 5954 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0ac0444-b127-4046-80b2-49f8d1a416e5-tmp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.416795 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416472 5954 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac0444-b127-4046-80b2-49f8d1a416e5-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.416795 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416481 5954 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0ac0444-b127-4046-80b2-49f8d1a416e5-console-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.416795 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.416490 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnh2\" (UniqueName: \"kubernetes.io/projected/a0ac0444-b127-4046-80b2-49f8d1a416e5-kube-api-access-qfnh2\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 19:59:35.958973 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.958932 5954 generic.go:356] "Generic (PLEG): container finished" podID="a0ac0444-b127-4046-80b2-49f8d1a416e5" containerID="b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb" exitCode=0 Nov 03 19:59:35.959169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.958989 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b794d6b84-mrbt9" Nov 03 19:59:35.959169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.958988 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b794d6b84-mrbt9" event={"ID":"a0ac0444-b127-4046-80b2-49f8d1a416e5","Type":"ContainerDied","Data":"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb"} Nov 03 19:59:35.959169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.959042 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b794d6b84-mrbt9" event={"ID":"a0ac0444-b127-4046-80b2-49f8d1a416e5","Type":"ContainerDied","Data":"2d8a303339b3e562f7c1bf1559fc625befab53bcf7c0d9fa80a5b9be7a52d1d2"} Nov 03 19:59:35.959169 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.959061 5954 scope.go:122] "RemoveContainer" containerID="b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb" Nov 03 19:59:35.967157 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.967138 5954 scope.go:122] "RemoveContainer" containerID="b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb" Nov 03 19:59:35.967411 ip-10-0-135-70 kubenswrapper[5954]: E1103 19:59:35.967393 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb\": container with ID starting with b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb not found: ID does not exist" containerID="b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb" Nov 03 19:59:35.967490 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.967422 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb"} err="failed to get container status \"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb\": rpc error: code = NotFound desc = could not find container \"b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb\": container with ID starting with b09e3d66e7a392d06bb9292033a39ba766f2ffae3e836387e88adf00e6b81fcb not found: ID does not exist" Nov 03 19:59:35.976481 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.976445 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:59:35.980141 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:35.980122 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b794d6b84-mrbt9"] Nov 03 19:59:37.401946 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:37.401909 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ac0444-b127-4046-80b2-49f8d1a416e5" path="/var/lib/kubelet/pods/a0ac0444-b127-4046-80b2-49f8d1a416e5/volumes" Nov 03 19:59:41.179889 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:41.179854 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:59:46.619241 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:46.619207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 19:59:57.186050 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:57.186006 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:59:57.201317 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:57.201295 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 19:59:58.054600 ip-10-0-135-70 kubenswrapper[5954]: I1103 19:59:58.054570 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:00:00.256697 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:00:00.256660 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:00:05.723339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:05.723302 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:05.827119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:05.827085 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:08.114559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:08.114516 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:08.212509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:08.212472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:11.735242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.735210 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d455578cf-qdn4g"] Nov 03 20:00:11.735642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.735556 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a0ac0444-b127-4046-80b2-49f8d1a416e5" containerName="console" Nov 03 20:00:11.735642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.735568 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ac0444-b127-4046-80b2-49f8d1a416e5" containerName="console" Nov 03 20:00:11.735642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.735628 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ac0444-b127-4046-80b2-49f8d1a416e5" containerName="console" Nov 03 20:00:11.739702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.739682 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.750945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.750913 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d455578cf-qdn4g"] Nov 03 20:00:11.834589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834560 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc757\" (UniqueName: \"kubernetes.io/projected/39df0b28-aa27-4fa6-a126-7c283627fd92-kube-api-access-kc757\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834595 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-console-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834614 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-oauth-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834634 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-service-ca\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834692 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834749 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39df0b28-aa27-4fa6-a126-7c283627fd92-tmp\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834782 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-trusted-ca-bundle\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.834906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.834804 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-oauth-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.847639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.847611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:11.936081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936050 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc757\" (UniqueName: \"kubernetes.io/projected/39df0b28-aa27-4fa6-a126-7c283627fd92-kube-api-access-kc757\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936086 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-console-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936102 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-oauth-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936138 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-service-ca\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936154 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936169 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39df0b28-aa27-4fa6-a126-7c283627fd92-tmp\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936205 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-trusted-ca-bundle\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936236 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-oauth-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936738 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39df0b28-aa27-4fa6-a126-7c283627fd92-tmp\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.936952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936925 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-console-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.937068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.936981 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-service-ca\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.937068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.937038 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-oauth-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.937068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.937056 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39df0b28-aa27-4fa6-a126-7c283627fd92-trusted-ca-bundle\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.938800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.938779 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-oauth-config\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.938858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.938781 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39df0b28-aa27-4fa6-a126-7c283627fd92-console-serving-cert\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:11.947587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.947560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:11.948677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:11.948656 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc757\" (UniqueName: \"kubernetes.io/projected/39df0b28-aa27-4fa6-a126-7c283627fd92-kube-api-access-kc757\") pod \"console-5d455578cf-qdn4g\" (UID: \"39df0b28-aa27-4fa6-a126-7c283627fd92\") " pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:12.051989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:12.051912 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:12.331655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:12.331563 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d455578cf-qdn4g"] Nov 03 20:00:12.336238 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:00:12.336211 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39df0b28_aa27_4fa6_a126_7c283627fd92.slice/crio-bca7aa4d113102b5c55e7de87c18ee49080716c0f2bcbbcaffb453f77c3e0ceb WatchSource:0}: Error finding container bca7aa4d113102b5c55e7de87c18ee49080716c0f2bcbbcaffb453f77c3e0ceb: Status 404 returned error can't find the container with id bca7aa4d113102b5c55e7de87c18ee49080716c0f2bcbbcaffb453f77c3e0ceb Nov 03 20:00:13.105531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:13.105491 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d455578cf-qdn4g" event={"ID":"39df0b28-aa27-4fa6-a126-7c283627fd92","Type":"ContainerStarted","Data":"70a00b83703b70ba19b99395aaac48be275aca62db25a83ed055f44628259915"} Nov 03 20:00:13.105955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:13.105576 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d455578cf-qdn4g" event={"ID":"39df0b28-aa27-4fa6-a126-7c283627fd92","Type":"ContainerStarted","Data":"bca7aa4d113102b5c55e7de87c18ee49080716c0f2bcbbcaffb453f77c3e0ceb"} Nov 03 20:00:13.127341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:13.127288 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d455578cf-qdn4g" podStartSLOduration=2.127273001 podStartE2EDuration="2.127273001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:13.126596757 +0000 UTC m=+153.851604353" watchObservedRunningTime="2026-11-03 20:00:13.127273001 +0000 UTC m=+153.852280599" Nov 03 20:00:13.579781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:13.579695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:13.679435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:13.679405 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:18.423463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:18.423431 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:18.523256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:18.523225 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:22.052637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:22.052603 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:22.053034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:22.052689 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:22.057657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:22.057634 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:22.140653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:22.140626 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d455578cf-qdn4g" Nov 03 20:00:22.190944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:22.190917 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 20:00:26.075045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:26.075003 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:26.175723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:26.175687 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:30.256305 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:00:30.256269 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:00:46.683736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:46.683667 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:47.211647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.211611 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b44ff6d88-q725z" podUID="dec41c4c-0dd0-4335-bab0-d603af234817" containerName="console" containerID="cri-o://6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b" gracePeriod=15 Nov 03 20:00:47.453291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.453268 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 20:00:47.551934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.551850 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.551934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.551897 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.551934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.551926 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.551956 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\" (UniqueName: \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.551986 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552020 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552127 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552156 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\") pod \"dec41c4c-0dd0-4335-bab0-d603af234817\" (UID: \"dec41c4c-0dd0-4335-bab0-d603af234817\") " Nov 03 20:00:47.552470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552414 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:00:47.552470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552441 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:47.552470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552446 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:47.552601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552517 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:47.552680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.552663 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:00:47.554359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.554335 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "kube-api-access-j2jzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:00:47.554359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.554332 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:47.554499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.554418 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config" pod "dec41c4c-0dd0-4335-bab0-d603af234817" (UID: "dec41c4c-0dd0-4335-bab0-d603af234817"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:00:47.653613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653584 5954 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-oauth-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653609 5954 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653623 5954 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-console-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653637 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jzf\" (UniqueName: \"kubernetes.io/projected/dec41c4c-0dd0-4335-bab0-d603af234817-kube-api-access-j2jzf\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653649 5954 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-serving-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653658 5954 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec41c4c-0dd0-4335-bab0-d603af234817-tmp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653666 5954 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec41c4c-0dd0-4335-bab0-d603af234817-console-oauth-config\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:47.653805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:47.653675 5954 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec41c4c-0dd0-4335-bab0-d603af234817-trusted-ca-bundle\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:00:48.235954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.235917 5954 generic.go:356] "Generic (PLEG): container finished" podID="dec41c4c-0dd0-4335-bab0-d603af234817" containerID="6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b" exitCode=0 Nov 03 20:00:48.236387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.235962 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b44ff6d88-q725z" Nov 03 20:00:48.236387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.236006 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b44ff6d88-q725z" event={"ID":"dec41c4c-0dd0-4335-bab0-d603af234817","Type":"ContainerDied","Data":"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b"} Nov 03 20:00:48.236387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.236058 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b44ff6d88-q725z" event={"ID":"dec41c4c-0dd0-4335-bab0-d603af234817","Type":"ContainerDied","Data":"6bc4993fa4c60967b46880d318376c8a3170d4bb25ff28f447389f85fbfb47d2"} Nov 03 20:00:48.236387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.236079 5954 scope.go:122] "RemoveContainer" containerID="6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b" Nov 03 20:00:48.244705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.244688 5954 scope.go:122] "RemoveContainer" containerID="6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b" Nov 03 20:00:48.244961 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:00:48.244944 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b\": container with ID starting with 6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b not found: ID does not exist" containerID="6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b" Nov 03 20:00:48.245007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.244970 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b"} err="failed to get container status \"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b\": rpc error: code = NotFound desc = could not find container \"6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b\": container with ID starting with 6ec7050368ae1a6e4bebdd6a587c55d0f8f768cfef000d88b5f9fe745ce1597b not found: ID does not exist" Nov 03 20:00:48.257206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.257185 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 20:00:48.261047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:48.261028 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b44ff6d88-q725z"] Nov 03 20:00:49.402413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:00:49.402381 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec41c4c-0dd0-4335-bab0-d603af234817" path="/var/lib/kubelet/pods/dec41c4c-0dd0-4335-bab0-d603af234817/volumes" Nov 03 20:01:00.256076 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:01:00.256038 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:01:30.258597 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:01:30.258548 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:01:46.726198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:01:46.726162 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:00.257318 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:02:00.257272 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:02:09.231008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:09.230972 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:10.093852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:10.093814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:18.910476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:18.910442 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:19.008853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:19.008820 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:30.256721 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:02:30.256635 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:02:44.927821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:44.927786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:45.026766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:45.026714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:46.782262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:46.782225 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:47.805370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:47.805335 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:47.908887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:02:47.908854 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:00.256819 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:03:00.256776 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:03:17.605014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:17.604982 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:17.735362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:17.735334 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:17.840332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:17.840302 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:19.084060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:19.084024 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:19.191994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:19.191958 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:20.370364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:20.370333 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:23.344822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:23.344791 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:23.445727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:23.445692 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:24.526235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:24.526199 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:24.624638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:24.624592 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:24.736334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:24.736302 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:27.293150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:27.293120 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:30.257085 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:03:30.257051 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:03:39.415955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.415924 5954 scope.go:122] "RemoveContainer" containerID="eec2ed1542d04fd24ffddc22553e3dd47bc7d254604dad49989e148a6853e9b8" Nov 03 20:03:39.423806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.423788 5954 scope.go:122] "RemoveContainer" containerID="f1caaa4597cd1d72fb8bba5c95b4de86e587a36dd42bdb9400b65c478d62bcd8" Nov 03 20:03:39.430934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.430916 5954 scope.go:122] "RemoveContainer" containerID="dadada5ebba3a4035f3e88d8407fac4175aca7617af99b5be01b2a474e8fb945" Nov 03 20:03:39.437908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.437893 5954 scope.go:122] "RemoveContainer" containerID="35718d8b301a82b15b3650cb427837a2c611b697b68c0ea4d80f82c3bdfd8cde" Nov 03 20:03:39.444287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.444248 5954 scope.go:122] "RemoveContainer" containerID="b6fa5b8aa2b08aff66e9a3fa686105c0c5c6d6db5dd905248769fdecea990e1e" Nov 03 20:03:39.450716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.450700 5954 scope.go:122] "RemoveContainer" containerID="3cd18385c076dc354a28253a87a6afad04c3437655dbd2161c8aa2ea5ef07d05" Nov 03 20:03:39.456833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:39.456818 5954 scope.go:122] "RemoveContainer" containerID="18ab646cae9c0d881559e7ba04f7f42edb2e03d81befcc89a85639d0188a3d24" Nov 03 20:03:46.838950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:46.838917 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:57.899238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:57.899202 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:57.998475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:03:57.998439 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:00.256385 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:04:00.256346 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:04:02.141988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:02.141955 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:02.240501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:02.240468 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:03.431609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.431578 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf"] Nov 03 20:04:03.432254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.431883 5954 cpu_manager.go:414] "containerMap: removing container" podUID="dec41c4c-0dd0-4335-bab0-d603af234817" containerName="console" Nov 03 20:04:03.432254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.431893 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec41c4c-0dd0-4335-bab0-d603af234817" containerName="console" Nov 03 20:04:03.432254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.431968 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec41c4c-0dd0-4335-bab0-d603af234817" containerName="console" Nov 03 20:04:03.434714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.434697 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.439504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.439487 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:04:03.439598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.439474 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:04:03.439771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.439754 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-sw62p\"" Nov 03 20:04:03.444422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.444403 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf"] Nov 03 20:04:03.521266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.521233 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff81d-ffde-414f-9d0c-e212a518f70f-tmp\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.521429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.521282 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qlw\" (UniqueName: \"kubernetes.io/projected/fbdff81d-ffde-414f-9d0c-e212a518f70f-kube-api-access-74qlw\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.622479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.622447 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff81d-ffde-414f-9d0c-e212a518f70f-tmp\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.622621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.622489 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qlw\" (UniqueName: \"kubernetes.io/projected/fbdff81d-ffde-414f-9d0c-e212a518f70f-kube-api-access-74qlw\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.622824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.622806 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbdff81d-ffde-414f-9d0c-e212a518f70f-tmp\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.634918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.634888 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qlw\" (UniqueName: \"kubernetes.io/projected/fbdff81d-ffde-414f-9d0c-e212a518f70f-kube-api-access-74qlw\") pod \"jobset-operator-674cd59dc5-kmhnf\" (UID: \"fbdff81d-ffde-414f-9d0c-e212a518f70f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:03.760074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:03.759997 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" Nov 03 20:04:04.031172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:04.031080 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf"] Nov 03 20:04:04.034749 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:04.034719 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbdff81d_ffde_414f_9d0c_e212a518f70f.slice/crio-071a90c33a5e003dd76e06f9a60184958963ddfb5eed800c69c467fd8f2a0dcd WatchSource:0}: Error finding container 071a90c33a5e003dd76e06f9a60184958963ddfb5eed800c69c467fd8f2a0dcd: Status 404 returned error can't find the container with id 071a90c33a5e003dd76e06f9a60184958963ddfb5eed800c69c467fd8f2a0dcd Nov 03 20:04:04.036173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:04.036157 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:04:04.956348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:04.956313 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" event={"ID":"fbdff81d-ffde-414f-9d0c-e212a518f70f","Type":"ContainerStarted","Data":"071a90c33a5e003dd76e06f9a60184958963ddfb5eed800c69c467fd8f2a0dcd"} Nov 03 20:04:07.969975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:07.969944 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" event={"ID":"fbdff81d-ffde-414f-9d0c-e212a518f70f","Type":"ContainerStarted","Data":"41b25327e97154d0ffefc99ac3c745e6199598f35afc09201b5b0e130f989e44"} Nov 03 20:04:07.990145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:07.990080 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kmhnf" podStartSLOduration=1.979473305 podStartE2EDuration="4.990066869s" totalImagesPullingTime="3.010593564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:07.989135662 +0000 UTC m=+388.714143262" watchObservedRunningTime="2026-11-03 20:04:07.990066869 +0000 UTC m=+388.715074464" Nov 03 20:04:08.006975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:08.006946 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:09.207747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:09.207716 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:11.872094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:11.872052 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:11.971742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:11.971696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:15.018800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:15.018750 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:16.637773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:16.637745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:16.854748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:16.854717 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:17.954067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:17.954030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:18.158558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.158525 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n"] Nov 03 20:04:18.162265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.162237 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.187232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.187204 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:04:18.187359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.187320 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:04:18.187944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.187928 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:04:18.188161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.188147 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:04:18.188366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.188352 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:04:18.188420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.188410 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t7ddc\"" Nov 03 20:04:18.197604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.197583 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n"] Nov 03 20:04:18.248762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248683 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.248762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248718 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.248931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248820 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.248931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248844 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.248931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248862 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8htt\" (UniqueName: \"kubernetes.io/projected/f46746bf-f978-4b56-8167-0a57befac650-kube-api-access-d8htt\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.249029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.248953 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.349925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.349890 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.349946 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.349968 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.350021 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.350044 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.350060 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8htt\" (UniqueName: \"kubernetes.io/projected/f46746bf-f978-4b56-8167-0a57befac650-kube-api-access-d8htt\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.350836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.350514 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.352912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.352881 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.353174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.353154 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.353385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.353364 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f46746bf-f978-4b56-8167-0a57befac650-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.353579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.353557 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f46746bf-f978-4b56-8167-0a57befac650-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.362360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.362335 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8htt\" (UniqueName: \"kubernetes.io/projected/f46746bf-f978-4b56-8167-0a57befac650-kube-api-access-d8htt\") pod \"opendatahub-operator-controller-manager-79746d584b-64r8n\" (UID: \"f46746bf-f978-4b56-8167-0a57befac650\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.377515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.377487 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:18.477002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.476961 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:18.486581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.486555 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:18.591549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.591513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:18.705012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.704979 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:18.787348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.787273 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n"] Nov 03 20:04:18.792140 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:18.792097 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46746bf_f978_4b56_8167_0a57befac650.slice/crio-c40853864d92e42641fa75defeba1854106cc863e1d739a75229ad8c9984f44c WatchSource:0}: Error finding container c40853864d92e42641fa75defeba1854106cc863e1d739a75229ad8c9984f44c: Status 404 returned error can't find the container with id c40853864d92e42641fa75defeba1854106cc863e1d739a75229ad8c9984f44c Nov 03 20:04:18.810615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:18.810591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:19.013273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:19.013239 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" event={"ID":"f46746bf-f978-4b56-8167-0a57befac650","Type":"ContainerStarted","Data":"c40853864d92e42641fa75defeba1854106cc863e1d739a75229ad8c9984f44c"} Nov 03 20:04:19.218465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:19.218428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:19.701928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:19.701896 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:19.818314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:19.818280 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:21.373616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:21.373581 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:21.476705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:21.476666 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:21.576971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:21.576934 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:23.034222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:23.034141 5954 generic.go:356] "Generic (PLEG): container finished" podID="f46746bf-f978-4b56-8167-0a57befac650" containerID="57ca4bbc3256b0c5bdda34e89bbe1e556561b923699abda126da6aefd64d4f1c" exitCode=0 Nov 03 20:04:23.034222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:23.034208 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" event={"ID":"f46746bf-f978-4b56-8167-0a57befac650","Type":"ContainerDied","Data":"57ca4bbc3256b0c5bdda34e89bbe1e556561b923699abda126da6aefd64d4f1c"} Nov 03 20:04:23.244808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:23.244771 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:24.039950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:24.039917 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" event={"ID":"f46746bf-f978-4b56-8167-0a57befac650","Type":"ContainerStarted","Data":"a1497f83a99c263f3403a7f38a02dae07868c7837f5b8feb6e4386c963410ff9"} Nov 03 20:04:24.040428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:24.040036 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:24.118306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:24.118254 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" podStartSLOduration=1.812016001 podStartE2EDuration="6.118239307s" totalImagesPullingTime="3.835120143s" totalInitContainerRuntime="471.103163ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:24.115949001 +0000 UTC m=+404.840956609" watchObservedRunningTime="2026-11-03 20:04:24.118239307 +0000 UTC m=+404.843246900" Nov 03 20:04:25.343711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:25.343675 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:25.464404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:25.464366 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:26.430834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:26.430802 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:27.053202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:27.053157 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:27.163320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:27.163286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:28.869790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:28.869753 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:29.750479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.750448 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b"] Nov 03 20:04:29.756846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.756823 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:29.765160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.764662 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:04:29.765680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.765656 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:04:29.765839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.765818 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:04:29.766993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.766971 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b"] Nov 03 20:04:29.767163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.767148 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4d8vf\"" Nov 03 20:04:29.855722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.855685 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2b9\" (UniqueName: \"kubernetes.io/projected/1bd00c68-f651-453e-b2e9-126a86d5ba11-kube-api-access-ss2b9\") pod \"dns-operator-controller-manager-8557cbdb84-drz9b\" (UID: \"1bd00c68-f651-453e-b2e9-126a86d5ba11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:29.956619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.956573 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2b9\" (UniqueName: \"kubernetes.io/projected/1bd00c68-f651-453e-b2e9-126a86d5ba11-kube-api-access-ss2b9\") pod \"dns-operator-controller-manager-8557cbdb84-drz9b\" (UID: \"1bd00c68-f651-453e-b2e9-126a86d5ba11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:29.972912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:29.972887 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2b9\" (UniqueName: \"kubernetes.io/projected/1bd00c68-f651-453e-b2e9-126a86d5ba11-kube-api-access-ss2b9\") pod \"dns-operator-controller-manager-8557cbdb84-drz9b\" (UID: \"1bd00c68-f651-453e-b2e9-126a86d5ba11\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:30.070872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:30.070787 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:30.256895 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:04:30.256858 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:04:30.347429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:30.347349 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b"] Nov 03 20:04:30.350502 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:30.350465 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd00c68_f651_453e_b2e9_126a86d5ba11.slice/crio-1e64e193034c076f343a818ea66721852a9fe6471bc0dfd4281d60d172b8f946 WatchSource:0}: Error finding container 1e64e193034c076f343a818ea66721852a9fe6471bc0dfd4281d60d172b8f946: Status 404 returned error can't find the container with id 1e64e193034c076f343a818ea66721852a9fe6471bc0dfd4281d60d172b8f946 Nov 03 20:04:31.071336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:31.071300 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" event={"ID":"1bd00c68-f651-453e-b2e9-126a86d5ba11","Type":"ContainerStarted","Data":"1e64e193034c076f343a818ea66721852a9fe6471bc0dfd4281d60d172b8f946"} Nov 03 20:04:31.479639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:31.479598 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:31.582289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:31.582250 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:31.816445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:31.816358 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:32.465165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:32.465132 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:32.642852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:32.642817 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:33.252024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.251990 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2"] Nov 03 20:04:33.255720 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.255694 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.267370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.267342 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2"] Nov 03 20:04:33.270477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.270458 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 20:04:33.270767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.270743 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 20:04:33.270767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.270837 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qcqxj\"" Nov 03 20:04:33.277838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.277813 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:33.387759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.387725 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5bc593-5e89-4d37-8539-b96830cae8a6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.387926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.387769 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e5bc593-5e89-4d37-8539-b96830cae8a6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.387926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.387854 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqr7f\" (UniqueName: \"kubernetes.io/projected/1e5bc593-5e89-4d37-8539-b96830cae8a6-kube-api-access-pqr7f\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.396442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.396404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:33.489233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.489206 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5bc593-5e89-4d37-8539-b96830cae8a6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.489533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.489246 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e5bc593-5e89-4d37-8539-b96830cae8a6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.489533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.489276 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqr7f\" (UniqueName: \"kubernetes.io/projected/1e5bc593-5e89-4d37-8539-b96830cae8a6-kube-api-access-pqr7f\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.489913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.489895 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e5bc593-5e89-4d37-8539-b96830cae8a6-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.491417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.491401 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e5bc593-5e89-4d37-8539-b96830cae8a6-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.506564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.506507 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqr7f\" (UniqueName: \"kubernetes.io/projected/1e5bc593-5e89-4d37-8539-b96830cae8a6-kube-api-access-pqr7f\") pod \"kuadrant-console-plugin-66c4fb575d-5nlk2\" (UID: \"1e5bc593-5e89-4d37-8539-b96830cae8a6\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.569907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.569873 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" Nov 03 20:04:33.842639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:33.842536 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2"] Nov 03 20:04:33.845568 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:33.845542 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5bc593_5e89_4d37_8539_b96830cae8a6.slice/crio-9cd30c8fe2acb82322835c696ca43631afd971aa54b3d307c8ba60e3739b672e WatchSource:0}: Error finding container 9cd30c8fe2acb82322835c696ca43631afd971aa54b3d307c8ba60e3739b672e: Status 404 returned error can't find the container with id 9cd30c8fe2acb82322835c696ca43631afd971aa54b3d307c8ba60e3739b672e Nov 03 20:04:34.087494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:34.087455 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" event={"ID":"1bd00c68-f651-453e-b2e9-126a86d5ba11","Type":"ContainerStarted","Data":"45967a40e5a8a5c8e05a28bf00fe10e7337139e9323f6c09368448d9b752cc88"} Nov 03 20:04:34.087650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:34.087572 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:34.088598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:34.088578 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" event={"ID":"1e5bc593-5e89-4d37-8539-b96830cae8a6","Type":"ContainerStarted","Data":"9cd30c8fe2acb82322835c696ca43631afd971aa54b3d307c8ba60e3739b672e"} Nov 03 20:04:34.111149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:34.111084 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" podStartSLOduration=2.050838198 podStartE2EDuration="5.111072615s" totalImagesPullingTime="3.060234417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:34.109838357 +0000 UTC m=+414.834845950" watchObservedRunningTime="2026-11-03 20:04:34.111072615 +0000 UTC m=+414.836080208" Nov 03 20:04:35.046033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:35.045998 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-64r8n" Nov 03 20:04:36.497036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:36.497002 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:38.051271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:38.051240 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:38.175761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:38.175728 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:38.693902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:38.693862 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:38.857027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:38.856993 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:39.118061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:39.118026 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" event={"ID":"1e5bc593-5e89-4d37-8539-b96830cae8a6","Type":"ContainerStarted","Data":"4d2e0940c13ee254b59147f455727a426aa183486abef941199631f4bf7d16f0"} Nov 03 20:04:39.138228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:39.138182 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5nlk2" podStartSLOduration=1.0766123890000001 podStartE2EDuration="6.138170254s" totalImagesPullingTime="5.061557865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:39.137818783 +0000 UTC m=+419.862826396" watchObservedRunningTime="2026-11-03 20:04:39.138170254 +0000 UTC m=+419.863177870" Nov 03 20:04:40.023073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.023042 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:40.166672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.166633 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:40.263126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.263082 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n"] Nov 03 20:04:40.266016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.265997 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.277767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.277715 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:04:40.277863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.277797 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:04:40.285949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.285918 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n"] Nov 03 20:04:40.359170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.359125 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.359343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.359296 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgqfb\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-kube-api-access-dgqfb\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.460669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.460633 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgqfb\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-kube-api-access-dgqfb\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.460872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.460694 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.474768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.474735 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.475925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.475901 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgqfb\" (UniqueName: \"kubernetes.io/projected/3602a4e5-7b74-44e0-a640-3e608c97bf19-kube-api-access-dgqfb\") pod \"cert-manager-webhook-5dbbbf75cc-64x5n\" (UID: \"3602a4e5-7b74-44e0-a640-3e608c97bf19\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.576511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.576426 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:40.856743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:40.856698 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n"] Nov 03 20:04:40.859733 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:40.859690 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3602a4e5_7b74_44e0_a640_3e608c97bf19.slice/crio-0e97eb0636637408a471504edad81047c54bb513cdbb95bf14bac85497f08f48 WatchSource:0}: Error finding container 0e97eb0636637408a471504edad81047c54bb513cdbb95bf14bac85497f08f48: Status 404 returned error can't find the container with id 0e97eb0636637408a471504edad81047c54bb513cdbb95bf14bac85497f08f48 Nov 03 20:04:41.128320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:41.128239 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" event={"ID":"3602a4e5-7b74-44e0-a640-3e608c97bf19","Type":"ContainerStarted","Data":"0e97eb0636637408a471504edad81047c54bb513cdbb95bf14bac85497f08f48"} Nov 03 20:04:42.369650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.369619 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:42.561936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.561906 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:42.743846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.742372 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd"] Nov 03 20:04:42.746786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.746760 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:42.753848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.753812 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sbss2\"" Nov 03 20:04:42.754865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.754832 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd"] Nov 03 20:04:42.881732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.881681 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9pj9\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-kube-api-access-x9pj9\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:42.881913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.881747 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:42.983136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.983080 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:42.983305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.983246 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9pj9\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-kube-api-access-x9pj9\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:42.998759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:42.997270 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9pj9\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-kube-api-access-x9pj9\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:43.002482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:43.002454 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/540c54d1-beab-4303-90af-20c33fed7c40-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-glfbd\" (UID: \"540c54d1-beab-4303-90af-20c33fed7c40\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:43.061906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:43.061877 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" Nov 03 20:04:43.339809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:43.339731 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd"] Nov 03 20:04:43.343058 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:43.343021 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod540c54d1_beab_4303_90af_20c33fed7c40.slice/crio-170fbeacb0a489150ec4f97cc873e62a3ea1e4b3c5f33a7ec25835ac93231cc7 WatchSource:0}: Error finding container 170fbeacb0a489150ec4f97cc873e62a3ea1e4b3c5f33a7ec25835ac93231cc7: Status 404 returned error can't find the container with id 170fbeacb0a489150ec4f97cc873e62a3ea1e4b3c5f33a7ec25835ac93231cc7 Nov 03 20:04:44.147898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:44.147852 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" event={"ID":"540c54d1-beab-4303-90af-20c33fed7c40","Type":"ContainerStarted","Data":"170fbeacb0a489150ec4f97cc873e62a3ea1e4b3c5f33a7ec25835ac93231cc7"} Nov 03 20:04:44.489936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:44.489903 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:44.643144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:44.643094 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:45.098839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.098811 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-drz9b" Nov 03 20:04:45.156089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.155887 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" event={"ID":"3602a4e5-7b74-44e0-a640-3e608c97bf19","Type":"ContainerStarted","Data":"66a8e4475740996fa3bc3445c68da475c0b9658f112b89832cc213b8b85b91f8"} Nov 03 20:04:45.156089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.155941 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:45.175498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.175443 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podStartSLOduration=1.6847710089999999 podStartE2EDuration="5.175424864s" totalImagesPullingTime="3.490653855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:45.174483189 +0000 UTC m=+425.899490811" watchObservedRunningTime="2026-11-03 20:04:45.175424864 +0000 UTC m=+425.900432464" Nov 03 20:04:45.674953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.674878 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:45.781665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:45.781630 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.025033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:46.024955 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.161547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:46.161513 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" event={"ID":"540c54d1-beab-4303-90af-20c33fed7c40","Type":"ContainerStarted","Data":"c4dbb412570219856337cbc342ed0ccf843f9f0c74cf25c33dac01318fbb485f"} Nov 03 20:04:46.192088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:46.192055 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:46.199206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:46.199151 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-glfbd" podStartSLOduration=2.223664742 podStartE2EDuration="4.199102355s" totalImagesPullingTime="1.975437613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:46.197463413 +0000 UTC m=+426.922471010" watchObservedRunningTime="2026-11-03 20:04:46.199102355 +0000 UTC m=+426.924109951" Nov 03 20:04:46.893062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:46.893029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:47.219367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:47.219284 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:47.320101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:47.320061 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:49.970707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:49.970671 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:51.164018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.163991 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" Nov 03 20:04:51.792130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.792081 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8ptkj"] Nov 03 20:04:51.795263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.795243 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:51.804626 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.804586 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wp8kp\"" Nov 03 20:04:51.805836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.805814 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8ptkj"] Nov 03 20:04:51.966745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.966711 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhm7\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-kube-api-access-8fhm7\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:51.966910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:51.966838 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-bound-sa-token\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.067494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.067414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-bound-sa-token\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.067494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.067471 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhm7\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-kube-api-access-8fhm7\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.080123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.080077 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-bound-sa-token\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.080241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.080216 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhm7\" (UniqueName: \"kubernetes.io/projected/5702e6c1-bf12-412c-82b8-39788ab111d5-kube-api-access-8fhm7\") pod \"cert-manager-597b95cb84-8ptkj\" (UID: \"5702e6c1-bf12-412c-82b8-39788ab111d5\") " pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.107027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.107000 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8ptkj" Nov 03 20:04:52.380684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.380656 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8ptkj"] Nov 03 20:04:52.384260 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:04:52.384230 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5702e6c1_bf12_412c_82b8_39788ab111d5.slice/crio-60d4deb4df43fad7874f826de497785974b698aab2ac12970d48b390990180c5 WatchSource:0}: Error finding container 60d4deb4df43fad7874f826de497785974b698aab2ac12970d48b390990180c5: Status 404 returned error can't find the container with id 60d4deb4df43fad7874f826de497785974b698aab2ac12970d48b390990180c5 Nov 03 20:04:52.814145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.814029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:52.907061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:52.907020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:53.193685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:53.193646 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8ptkj" event={"ID":"5702e6c1-bf12-412c-82b8-39788ab111d5","Type":"ContainerStarted","Data":"118b6623bb84add68569543af087e1f10689ebd736f7c07d93eb0b7087807c7e"} Nov 03 20:04:53.193864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:53.193698 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8ptkj" event={"ID":"5702e6c1-bf12-412c-82b8-39788ab111d5","Type":"ContainerStarted","Data":"60d4deb4df43fad7874f826de497785974b698aab2ac12970d48b390990180c5"} Nov 03 20:04:53.215029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:53.214975 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8ptkj" podStartSLOduration=2.214959488 podStartE2EDuration="2.214959488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:53.214485633 +0000 UTC m=+433.939493230" watchObservedRunningTime="2026-11-03 20:04:53.214959488 +0000 UTC m=+433.939967084" Nov 03 20:04:54.598090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:54.598053 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:57.614825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:57.614782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:57.714845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:57.714813 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:58.210961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:58.210926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:58.306008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:04:58.305970 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:00.256982 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:05:00.256939 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:05:00.692542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:00.692506 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:00.793571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:00.793535 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:00.972324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:00.972247 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:00.997903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:00.997867 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-vncsb"] Nov 03 20:05:01.001310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.001293 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.018285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.018256 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:05:01.018671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.018426 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:05:01.019592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.019562 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-vncsb"] Nov 03 20:05:01.022568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.022545 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:05:01.022693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.022584 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jkmgq\"" Nov 03 20:05:01.022693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.022644 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:05:01.037709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.037679 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tmp\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.037846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.037739 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hrr\" (UniqueName: \"kubernetes.io/projected/6e045ad4-d958-4322-814b-d2c57c8d8cfd-kube-api-access-s6hrr\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.037846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.037822 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tls-certs\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.138764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.138718 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tmp\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.138916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.138771 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hrr\" (UniqueName: \"kubernetes.io/projected/6e045ad4-d958-4322-814b-d2c57c8d8cfd-kube-api-access-s6hrr\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.138976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.138936 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tls-certs\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.141181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.141141 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:01.141409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.141383 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tmp\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.141747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.141726 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e045ad4-d958-4322-814b-d2c57c8d8cfd-tls-certs\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.164471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.164436 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hrr\" (UniqueName: \"kubernetes.io/projected/6e045ad4-d958-4322-814b-d2c57c8d8cfd-kube-api-access-s6hrr\") pod \"kube-auth-proxy-667774d4b7-vncsb\" (UID: \"6e045ad4-d958-4322-814b-d2c57c8d8cfd\") " pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.252190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.252074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:01.314291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.314256 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" Nov 03 20:05:01.354133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.354040 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:01.588286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.588190 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-667774d4b7-vncsb"] Nov 03 20:05:01.591130 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:05:01.591083 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e045ad4_d958_4322_814b_d2c57c8d8cfd.slice/crio-af4db7558409f0f59c659ee25b0c3c5faec94dc32181d445bcd186cae2d5f47f WatchSource:0}: Error finding container af4db7558409f0f59c659ee25b0c3c5faec94dc32181d445bcd186cae2d5f47f: Status 404 returned error can't find the container with id af4db7558409f0f59c659ee25b0c3c5faec94dc32181d445bcd186cae2d5f47f Nov 03 20:05:01.725240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.725208 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:01.835729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.835695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:01.936414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:01.936374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:02.234911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.234790 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" event={"ID":"6e045ad4-d958-4322-814b-d2c57c8d8cfd","Type":"ContainerStarted","Data":"af4db7558409f0f59c659ee25b0c3c5faec94dc32181d445bcd186cae2d5f47f"} Nov 03 20:05:02.452776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.452742 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:05:02.456573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.456539 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.462665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.462624 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:05:02.467660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.467632 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:05:02.497900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.497811 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:05:02.554006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.553963 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvlb\" (UniqueName: \"kubernetes.io/projected/1f257ac6-b47e-4fa8-80cd-66c2e263b816-kube-api-access-qcvlb\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.554233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.554185 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f257ac6-b47e-4fa8-80cd-66c2e263b816-config-file\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.655545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.655505 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f257ac6-b47e-4fa8-80cd-66c2e263b816-config-file\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.655735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.655584 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvlb\" (UniqueName: \"kubernetes.io/projected/1f257ac6-b47e-4fa8-80cd-66c2e263b816-kube-api-access-qcvlb\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.656288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.656260 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f257ac6-b47e-4fa8-80cd-66c2e263b816-config-file\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.668609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.668574 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvlb\" (UniqueName: \"kubernetes.io/projected/1f257ac6-b47e-4fa8-80cd-66c2e263b816-kube-api-access-qcvlb\") pod \"limitador-limitador-7c9b87d574-spzh9\" (UID: \"1f257ac6-b47e-4fa8-80cd-66c2e263b816\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.669343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.669314 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:02.774134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.774030 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:02.944444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:02.944371 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:03.053212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.053098 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:03.062066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.062033 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:05:03.069739 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:05:03.069698 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f257ac6_b47e_4fa8_80cd_66c2e263b816.slice/crio-91af7f3bbda28857e2143291b9eada951dd2b582000a4c42752be3d71150ec86 WatchSource:0}: Error finding container 91af7f3bbda28857e2143291b9eada951dd2b582000a4c42752be3d71150ec86: Status 404 returned error can't find the container with id 91af7f3bbda28857e2143291b9eada951dd2b582000a4c42752be3d71150ec86 Nov 03 20:05:03.150131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.150083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:03.242015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.241970 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" event={"ID":"1f257ac6-b47e-4fa8-80cd-66c2e263b816","Type":"ContainerStarted","Data":"91af7f3bbda28857e2143291b9eada951dd2b582000a4c42752be3d71150ec86"} Nov 03 20:05:03.423468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.423434 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:03.542179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:03.542098 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:04.249575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:04.249434 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" event={"ID":"6e045ad4-d958-4322-814b-d2c57c8d8cfd","Type":"ContainerStarted","Data":"398209f8be0822bddfa3c1bf1c24af55416301ca980680a29702908ef2969a9b"} Nov 03 20:05:04.268796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:04.268739 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-667774d4b7-vncsb" podStartSLOduration=1.8790478830000001 podStartE2EDuration="4.268723731s" totalImagesPullingTime="2.389675848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:04.268004325 +0000 UTC m=+444.993011929" watchObservedRunningTime="2026-11-03 20:05:04.268723731 +0000 UTC m=+444.993731390" Nov 03 20:05:05.260734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:05.260693 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" event={"ID":"1f257ac6-b47e-4fa8-80cd-66c2e263b816","Type":"ContainerStarted","Data":"6274e9edf1132628cc8a51dcddba5529c132cee917d96e960374e704901e1708"} Nov 03 20:05:05.281214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:05.281156 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" podStartSLOduration=1.83715045 podStartE2EDuration="3.28114273s" totalImagesPullingTime="1.44399228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:05.280269521 +0000 UTC m=+446.005277118" watchObservedRunningTime="2026-11-03 20:05:05.28114273 +0000 UTC m=+446.006150326" Nov 03 20:05:06.265100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:06.265064 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:09.024779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:09.024749 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:09.117433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:09.117398 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:09.618901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:09.618868 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:09.719954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:09.719924 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:10.222715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:10.222681 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:10.319662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:10.319624 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:11.469680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:11.469642 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:11.569289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:11.569257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:11.744725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:11.744642 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:11.838527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:11.838495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:15.507404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:15.507359 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:17.270510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:17.270482 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-spzh9" Nov 03 20:05:18.620453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:18.620418 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:18.731231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:18.731185 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:23.444558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:23.444458 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:23.544007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:23.543952 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:24.162391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:24.162359 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:30.257542 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:05:30.257495 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:05:33.101075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:33.100994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:33.217161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:33.217130 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:44.084774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:44.084736 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:44.186976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:44.186941 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:44.287303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:44.287265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:44.383169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:44.383099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:46.944925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:46.944894 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:47.964820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:47.964786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:48.065242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:48.065210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:48.973029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:48.972997 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:49.180129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:49.180079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.401492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.401457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.501538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.501509 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvd4x"] Nov 03 20:05:57.505150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.505132 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.511768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.511733 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.514542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.514488 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:05:57.514796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.514774 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 20:05:57.514891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.514781 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:05:57.516245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.516226 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p9z6r\"" Nov 03 20:05:57.516322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.516225 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvd4x"] Nov 03 20:05:57.549678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.549643 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9xf\" (UniqueName: \"kubernetes.io/projected/71d10093-69e4-4465-bc5c-4685a95d9427-kube-api-access-2n9xf\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.549804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.549701 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d10093-69e4-4465-bc5c-4685a95d9427-cert\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.565827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.565793 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-lv9j5"] Nov 03 20:05:57.569360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.569338 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.579910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.579886 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lv9j5"] Nov 03 20:05:57.599149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.599046 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 20:05:57.599442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.599424 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rj5wc\"" Nov 03 20:05:57.599552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.599489 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 20:05:57.610460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.610436 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.651192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.651163 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpjnf\" (UniqueName: \"kubernetes.io/projected/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-kube-api-access-bpjnf\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.651357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.651204 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-cert\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.651357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.651309 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9xf\" (UniqueName: \"kubernetes.io/projected/71d10093-69e4-4465-bc5c-4685a95d9427-kube-api-access-2n9xf\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.651469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.651402 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-metrics-certs\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.651469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.651438 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d10093-69e4-4465-bc5c-4685a95d9427-cert\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.654114 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.654045 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d10093-69e4-4465-bc5c-4685a95d9427-cert\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.664606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.664576 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9xf\" (UniqueName: \"kubernetes.io/projected/71d10093-69e4-4465-bc5c-4685a95d9427-kube-api-access-2n9xf\") pod \"kserve-controller-manager-776cc66d47-hvd4x\" (UID: \"71d10093-69e4-4465-bc5c-4685a95d9427\") " pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.718797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.718765 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.752062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.752031 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpjnf\" (UniqueName: \"kubernetes.io/projected/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-kube-api-access-bpjnf\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.752219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.752072 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-cert\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.752219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.752137 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-metrics-certs\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.754515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.754490 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-cert\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.754603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.754517 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-metrics-certs\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.763937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.763915 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpjnf\" (UniqueName: \"kubernetes.io/projected/694e6ee4-c04f-48ca-894f-1b0c4a63a7b5-kube-api-access-bpjnf\") pod \"odh-model-controller-769875db56-lv9j5\" (UID: \"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5\") " pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.818722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.818694 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:05:57.819718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.819691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.882851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.882750 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:05:57.907872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.907812 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h"] Nov 03 20:05:57.912445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.912425 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.916999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.916975 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:05:57.917140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.917007 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:05:57.917140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.917041 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:05:57.917140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.917072 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:05:57.917249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.917154 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xj9dq\"" Nov 03 20:05:57.925247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.925180 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h"] Nov 03 20:05:57.954728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954700 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74m2\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-kube-api-access-r74m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.954860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954753 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.954860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954828 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.954940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954881 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.954940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954928 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29ac0d4-d574-4773-8d82-81f822379713-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.955018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.954949 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:57.955126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:57.955082 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055518 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29ac0d4-d574-4773-8d82-81f822379713-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055551 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055597 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055628 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74m2\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-kube-api-access-r74m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055681 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055703 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.055752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.055721 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.056620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.056580 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.057819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.057794 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29ac0d4-d574-4773-8d82-81f822379713-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.057910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.057844 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.057989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.057918 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.058049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.058025 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f29ac0d4-d574-4773-8d82-81f822379713-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.072528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.072505 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.072801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.072782 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74m2\" (UniqueName: \"kubernetes.io/projected/f29ac0d4-d574-4773-8d82-81f822379713-kube-api-access-r74m2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bzp6h\" (UID: \"f29ac0d4-d574-4773-8d82-81f822379713\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.090874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.090850 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-hvd4x"] Nov 03 20:05:58.094238 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:05:58.094214 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d10093_69e4_4465_bc5c_4685a95d9427.slice/crio-df2113b926e3b64f5cd7b5c68089e52a5b7685ceae5eed2d3ca48d21cfb6b79c WatchSource:0}: Error finding container df2113b926e3b64f5cd7b5c68089e52a5b7685ceae5eed2d3ca48d21cfb6b79c: Status 404 returned error can't find the container with id df2113b926e3b64f5cd7b5c68089e52a5b7685ceae5eed2d3ca48d21cfb6b79c Nov 03 20:05:58.165340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.163578 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-lv9j5"] Nov 03 20:05:58.166773 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:05:58.166740 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694e6ee4_c04f_48ca_894f_1b0c4a63a7b5.slice/crio-266e3ddf3d92e72fb6392a91a0dcf9836f5356b71cfc6d99e19daa6d6f703216 WatchSource:0}: Error finding container 266e3ddf3d92e72fb6392a91a0dcf9836f5356b71cfc6d99e19daa6d6f703216: Status 404 returned error can't find the container with id 266e3ddf3d92e72fb6392a91a0dcf9836f5356b71cfc6d99e19daa6d6f703216 Nov 03 20:05:58.233382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.233352 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:05:58.501759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.501674 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" event={"ID":"71d10093-69e4-4465-bc5c-4685a95d9427","Type":"ContainerStarted","Data":"df2113b926e3b64f5cd7b5c68089e52a5b7685ceae5eed2d3ca48d21cfb6b79c"} Nov 03 20:05:58.502817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.502791 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lv9j5" event={"ID":"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5","Type":"ContainerStarted","Data":"266e3ddf3d92e72fb6392a91a0dcf9836f5356b71cfc6d99e19daa6d6f703216"} Nov 03 20:05:58.517277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.517058 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h"] Nov 03 20:05:58.519702 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:05:58.519677 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29ac0d4_d574_4773_8d82_81f822379713.slice/crio-b0ab8917aceacbc0bf11d65e2a3274935ff4535109de93c29903291893fedc40 WatchSource:0}: Error finding container b0ab8917aceacbc0bf11d65e2a3274935ff4535109de93c29903291893fedc40: Status 404 returned error can't find the container with id b0ab8917aceacbc0bf11d65e2a3274935ff4535109de93c29903291893fedc40 Nov 03 20:05:58.667258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.667213 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:58.764619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.764526 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:58.865024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.864940 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:58.995634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:58.995514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:59.513214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:05:59.513179 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" event={"ID":"f29ac0d4-d574-4773-8d82-81f822379713","Type":"ContainerStarted","Data":"b0ab8917aceacbc0bf11d65e2a3274935ff4535109de93c29903291893fedc40"} Nov 03 20:06:00.273439 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:06:00.273402 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:06:01.526985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.526955 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" event={"ID":"71d10093-69e4-4465-bc5c-4685a95d9427","Type":"ContainerStarted","Data":"a62812f9dc377b8c486c1bdc73470204ddf09509c027336a0056cfbd0683ddda"} Nov 03 20:06:01.527386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.527096 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:06:01.529467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.529424 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-lv9j5" event={"ID":"694e6ee4-c04f-48ca-894f-1b0c4a63a7b5","Type":"ContainerStarted","Data":"7831878b8fd19b540a312c54cf63feda3c4ade0a0783eaf55b29a0819ccec56f"} Nov 03 20:06:01.529596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.529514 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:06:01.547061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.547009 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" podStartSLOduration=1.906957682 podStartE2EDuration="4.546995605s" totalImagesPullingTime="2.640037923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:01.545470781 +0000 UTC m=+502.270478378" watchObservedRunningTime="2026-11-03 20:06:01.546995605 +0000 UTC m=+502.272003243" Nov 03 20:06:01.566248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.566199 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-lv9j5" podStartSLOduration=1.283342505 podStartE2EDuration="4.566183976s" totalImagesPullingTime="3.282841471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:01.564976729 +0000 UTC m=+502.289984350" watchObservedRunningTime="2026-11-03 20:06:01.566183976 +0000 UTC m=+502.291191572" Nov 03 20:06:01.792216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.792142 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:06:01.792216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:01.792206 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:06:02.535772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:02.535722 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" event={"ID":"f29ac0d4-d574-4773-8d82-81f822379713","Type":"ContainerStarted","Data":"2f2ca1bab5ff12850599c04d2d35ecc3371e6f57a5de3f0bd789931868472366"} Nov 03 20:06:02.536313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:02.536086 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:06:02.537656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:02.537626 5954 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bzp6h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:06:02.537758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:02.537679 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" podUID="f29ac0d4-d574-4773-8d82-81f822379713" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:06:02.561077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:02.561035 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" podStartSLOduration=2.2914355029999998 podStartE2EDuration="5.561023265s" totalImagesPullingTime="3.269587762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:02.56028119 +0000 UTC m=+503.285288787" watchObservedRunningTime="2026-11-03 20:06:02.561023265 +0000 UTC m=+503.286030860" Nov 03 20:06:03.547947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:03.547904 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bzp6h" Nov 03 20:06:12.538244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:12.538212 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-lv9j5" Nov 03 20:06:20.177796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:20.177759 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:20.281027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:20.280990 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:21.481831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:21.481804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:21.584820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:21.584786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:22.438779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:22.438743 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:06:30.260235 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:06:30.260088 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:06:31.955169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:31.955134 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:32.063235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:32.063205 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:32.541093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:32.541061 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-hvd4x" Nov 03 20:06:47.047533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:06:47.047501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:00.258098 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:07:00.258059 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:07:29.515012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:29.514978 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:30.257324 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:07:30.257290 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:07:30.985167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:30.985135 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:39.311631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:39.311595 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:40.157709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:40.157674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:47.049996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:47.049964 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:48.908066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:48.908022 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:49.437846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:49.437807 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:07:50.226533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:50.226502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:50.667834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:50.667803 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:07:50.974561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:50.974455 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:51.353737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:51.353683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:51.824313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:51.824230 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:52.537518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:52.537491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:53.446238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:53.446205 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:07:53.522080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:53.522045 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:56.060323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:56.060293 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:07:56.147311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:56.147276 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:56.944354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:56.944319 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:07:57.101849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:57.101815 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:58.689971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:07:58.689940 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:00.257627 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:08:00.257592 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:08:01.063671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.063634 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:08:01.374236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.374200 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl"] Nov 03 20:08:01.377811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.377787 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.384142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.384102 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 20:08:01.387398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.386962 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vbjn\"" Nov 03 20:08:01.387398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.386945 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:08:01.388056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.387947 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:08:01.390182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.390160 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl"] Nov 03 20:08:01.485515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.485482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:01.502245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502212 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930811f6-8cc8-4139-900f-7b4383999138-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.502401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502258 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.502401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502334 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.502472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502401 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.502526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502508 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.502577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.502563 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqjn\" (UniqueName: \"kubernetes.io/projected/930811f6-8cc8-4139-900f-7b4383999138-kube-api-access-nsqjn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603222 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930811f6-8cc8-4139-900f-7b4383999138-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603267 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603292 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603329 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603364 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603381 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqjn\" (UniqueName: \"kubernetes.io/projected/930811f6-8cc8-4139-900f-7b4383999138-kube-api-access-nsqjn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603747 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.603925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.603827 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.605514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.605492 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930811f6-8cc8-4139-900f-7b4383999138-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.605769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.605751 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930811f6-8cc8-4139-900f-7b4383999138-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.615953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.615927 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqjn\" (UniqueName: \"kubernetes.io/projected/930811f6-8cc8-4139-900f-7b4383999138-kube-api-access-nsqjn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl\" (UID: \"930811f6-8cc8-4139-900f-7b4383999138\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.691740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.691660 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:01.970666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:01.970637 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl"] Nov 03 20:08:01.973244 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:01.973221 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930811f6_8cc8_4139_900f_7b4383999138.slice/crio-a8cf17512511754af492cac83134dcb7eee8261e076aec2bf822dcd3f5849a70 WatchSource:0}: Error finding container a8cf17512511754af492cac83134dcb7eee8261e076aec2bf822dcd3f5849a70: Status 404 returned error can't find the container with id a8cf17512511754af492cac83134dcb7eee8261e076aec2bf822dcd3f5849a70 Nov 03 20:08:02.089355 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:02.089317 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" event={"ID":"930811f6-8cc8-4139-900f-7b4383999138","Type":"ContainerStarted","Data":"a8cf17512511754af492cac83134dcb7eee8261e076aec2bf822dcd3f5849a70"} Nov 03 20:08:04.102010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:04.101966 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" event={"ID":"930811f6-8cc8-4139-900f-7b4383999138","Type":"ContainerStarted","Data":"2fb3fef4b56cc9e3ae32a15578f24e6bf7a182617a0fdf6838deaab9c7c83b8a"} Nov 03 20:08:04.123275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:04.123226 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" podStartSLOduration=1.871968588 podStartE2EDuration="3.12321211s" totalImagesPullingTime="1.251243522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:04.12213444 +0000 UTC m=+624.847142037" watchObservedRunningTime="2026-11-03 20:08:04.12321211 +0000 UTC m=+624.848219707" Nov 03 20:08:04.691962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:04.691930 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:05.339434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:05.339399 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:08:09.847910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.847874 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:08:09.851708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.851685 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.855673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.855645 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwd2g\"" Nov 03 20:08:09.863813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.863786 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:08:09.878544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878510 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878553 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878578 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878647 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878746 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878799 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878844 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.878957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878930 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.879010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.878987 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2c2\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980364 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980413 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980470 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980521 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980571 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980628 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980663 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980749 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2c2\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.980845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.980783 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.981297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.981273 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.981455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.981318 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.981455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.981351 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.981592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.981367 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.981592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.981501 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.982915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.982892 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.983377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.983362 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.993499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.993466 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:09.993660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:09.993603 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2c2\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\") pod \"maas-default-gateway-openshift-default-77f556d647-gl4rp\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:10.165447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:10.165413 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:10.450192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:10.450162 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:08:10.452737 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:10.452711 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605626b1_de96_4f7e_83de_02621bca7bbc.slice/crio-91183893c98bb157a75105b561f46e24262da2b6cc582b1d86a3d5bf73410d80 WatchSource:0}: Error finding container 91183893c98bb157a75105b561f46e24262da2b6cc582b1d86a3d5bf73410d80: Status 404 returned error can't find the container with id 91183893c98bb157a75105b561f46e24262da2b6cc582b1d86a3d5bf73410d80 Nov 03 20:08:11.134328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:11.134289 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" event={"ID":"605626b1-de96-4f7e-83de-02621bca7bbc","Type":"ContainerStarted","Data":"91183893c98bb157a75105b561f46e24262da2b6cc582b1d86a3d5bf73410d80"} Nov 03 20:08:11.691927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:11.691892 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:11.704248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:11.704224 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:12.150482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:12.150457 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" Nov 03 20:08:14.305226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:14.305195 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:14.305484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:14.305266 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:14.305484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:14.305296 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:15.156796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:15.156763 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" event={"ID":"605626b1-de96-4f7e-83de-02621bca7bbc","Type":"ContainerStarted","Data":"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b"} Nov 03 20:08:15.165640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:15.165619 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:15.167122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:15.167080 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:15.167224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:15.167141 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:15.182656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:15.182603 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podStartSLOduration=2.333081984 podStartE2EDuration="6.182587544s" totalImagesPullingTime="3.84950556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:15.18228047 +0000 UTC m=+635.907288067" watchObservedRunningTime="2026-11-03 20:08:15.182587544 +0000 UTC m=+635.907595140" Nov 03 20:08:16.168174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:16.168138 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:16.168554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:16.168201 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:17.166547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:17.166510 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:17.166738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:17.166573 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:18.166049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:18.166010 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:18.166459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:18.166087 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:19.165919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:19.165883 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:19.166101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:19.165961 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:20.166276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:20.166199 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:20.166276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:20.166253 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:21.166155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:21.166119 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 20:08:21.166345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:21.166172 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 20:08:22.510234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:22.510193 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:22.510696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:22.510659 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:22.511464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:22.511445 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:08:24.883155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.883091 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:08:24.886810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.886785 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.899959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.899935 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:08:24.938657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.938624 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.938657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.938664 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.938900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.938687 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.938900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.938860 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.939015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.938906 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.939070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.939042 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.939131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.939085 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.939184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.939126 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:24.939184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:24.939152 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thklp\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040513 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040551 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040620 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040639 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040666 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040702 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thklp\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040736 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040765 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.040827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.040798 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.041223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.041082 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.041223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.041162 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.041223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.041203 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.041373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.041240 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.041668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.041647 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.042965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.042945 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.043270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.043254 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.053299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.053272 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.053423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.053406 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thklp\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\") pod \"maas-default-gateway-openshift-default-77f556d647-khb7v\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.202351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.202261 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:25.483359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.483333 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:08:25.485521 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:25.485489 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf6bb35_6c73_4e77_ab15_9eaa15b60ba0.slice/crio-11a190f968287562449f031c2be4a6b1c575c52805a3a943bb459685bf0ec05f WatchSource:0}: Error finding container 11a190f968287562449f031c2be4a6b1c575c52805a3a943bb459685bf0ec05f: Status 404 returned error can't find the container with id 11a190f968287562449f031c2be4a6b1c575c52805a3a943bb459685bf0ec05f Nov 03 20:08:25.488467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.488439 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:25.488549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.488500 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:25.488549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:25.488531 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:26.238695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:26.238656 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" event={"ID":"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0","Type":"ContainerStarted","Data":"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd"} Nov 03 20:08:26.239255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:26.238713 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" event={"ID":"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0","Type":"ContainerStarted","Data":"11a190f968287562449f031c2be4a6b1c575c52805a3a943bb459685bf0ec05f"} Nov 03 20:08:27.202671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:27.202642 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:27.204063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:27.204037 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:27.204205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:27.204085 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:28.202694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:28.202659 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:28.203089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:28.202718 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:29.203096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:29.203061 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:29.203492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:29.203142 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:30.203371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:30.203332 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:30.203846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:30.203394 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:30.263328 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:08:30.263292 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:08:31.202624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:31.202595 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:31.202782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:31.202645 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:32.203392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:32.203364 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 20:08:32.203741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:32.203413 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 20:08:33.676672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:33.676641 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:33.677272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:33.677073 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:33.678236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:33.678219 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:08:33.702121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:33.702065 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podStartSLOduration=9.702054118 podStartE2EDuration="9.702054118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:26.263787002 +0000 UTC m=+646.988794601" watchObservedRunningTime="2026-11-03 20:08:33.702054118 +0000 UTC m=+654.427061715" Nov 03 20:08:39.725812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:39.725777 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:08:39.922880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:39.922834 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:08:39.936351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:39.936315 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:39.940795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:39.940755 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:08:40.089148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089029 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089093 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxct\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089306 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089368 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089418 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089458 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089506 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089573 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.089705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.089604 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190322 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190380 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190400 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190541 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190604 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190636 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190709 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190754 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxct\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190836 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.190866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190857 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.191076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190900 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.191076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.190951 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.191264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.191246 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.191680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.191644 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.193098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.193073 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.193232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.193213 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.202605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.202567 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.202731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.202636 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxct\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\") pod \"maas-default-gateway-openshift-default-77f556d647-29vkz\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.253433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.253394 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:40.541774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.541638 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:08:40.544574 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:40.544538 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c47013_4463_4edd_b130_c4b93fea7af9.slice/crio-1ef88e4b6ef8e031ea5c2ec6d4735cc5ea178f6b32bc9fd1cf082b6aced3193a WatchSource:0}: Error finding container 1ef88e4b6ef8e031ea5c2ec6d4735cc5ea178f6b32bc9fd1cf082b6aced3193a: Status 404 returned error can't find the container with id 1ef88e4b6ef8e031ea5c2ec6d4735cc5ea178f6b32bc9fd1cf082b6aced3193a Nov 03 20:08:40.547636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.547596 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.547739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.547719 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.547787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:40.547771 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:41.320941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.320898 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" event={"ID":"40c47013-4463-4edd-b130-c4b93fea7af9","Type":"ContainerStarted","Data":"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d"} Nov 03 20:08:41.321578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.321553 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" event={"ID":"40c47013-4463-4edd-b130-c4b93fea7af9","Type":"ContainerStarted","Data":"1ef88e4b6ef8e031ea5c2ec6d4735cc5ea178f6b32bc9fd1cf082b6aced3193a"} Nov 03 20:08:41.388806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.388726 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podStartSLOduration=2.38870595 podStartE2EDuration="2.38870595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.384669078 +0000 UTC m=+662.109676677" watchObservedRunningTime="2026-11-03 20:08:41.38870595 +0000 UTC m=+662.113713547" Nov 03 20:08:41.832790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.832747 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:08:41.869129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.853074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:41.869622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.863919 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:41.885196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.863956 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:08:41.886960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.885721 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-m9thq\"" Nov 03 20:08:41.886960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.886447 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:08:41.886960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.886522 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:08:41.944131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.939462 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:41.944131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:41.939614 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkvf\" (UniqueName: \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.044706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.043727 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.044706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.043871 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkvf\" (UniqueName: \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.044706 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:08:42.044389 5954 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found Nov 03 20:08:42.044706 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:08:42.044469 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs podName:4b20221b-d7d0-4515-847d-02579b0c867a nodeName:}" failed. No retries permitted until 2026-11-03 20:08:42.544446513 +0000 UTC m=+663.269454108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs") pod "ai-gateway-controller-86c7698f44-clnb5" (UID: "4b20221b-d7d0-4515-847d-02579b0c867a") : secret "ai-gateway-controller-metrics-tls" not found Nov 03 20:08:42.079165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.070239 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkvf\" (UniqueName: \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.261130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.258541 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:42.269131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.268249 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:42.269131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.268316 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:42.392092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.391398 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:42.555627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.554790 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.560414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.560331 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") pod \"ai-gateway-controller-86c7698f44-clnb5\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:42.846413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:42.846322 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:43.132131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:43.130786 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:08:43.141384 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:43.141348 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b20221b_d7d0_4515_847d_02579b0c867a.slice/crio-fa17699d0b3153597f36f5399140815a540cd31227df1cc98f61a010a065a316 WatchSource:0}: Error finding container fa17699d0b3153597f36f5399140815a540cd31227df1cc98f61a010a065a316: Status 404 returned error can't find the container with id fa17699d0b3153597f36f5399140815a540cd31227df1cc98f61a010a065a316 Nov 03 20:08:43.254256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:43.254222 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:43.254444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:43.254277 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:43.344959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:43.344921 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" event={"ID":"4b20221b-d7d0-4515-847d-02579b0c867a","Type":"ContainerStarted","Data":"fa17699d0b3153597f36f5399140815a540cd31227df1cc98f61a010a065a316"} Nov 03 20:08:44.254344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:44.254299 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:44.254738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:44.254357 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:45.254333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:45.254285 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:45.254594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:45.254353 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:46.254490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:46.254446 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:46.254680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:46.254510 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:47.110455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.110417 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:47.254654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.254618 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:08:47.254846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.254683 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 20:08:47.291284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.291237 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:47.355202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.355162 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h4h49"] Nov 03 20:08:47.359692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.359670 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.370328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.370300 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:08:47.374690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.374666 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wgb65\"" Nov 03 20:08:47.380450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.380421 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h4h49"] Nov 03 20:08:47.384669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.384639 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" event={"ID":"4b20221b-d7d0-4515-847d-02579b0c867a","Type":"ContainerStarted","Data":"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d"} Nov 03 20:08:47.384816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.384783 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:47.414932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.414897 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:47.514378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.514340 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5nx\" (UniqueName: \"kubernetes.io/projected/3126a1b2-1b19-441d-912d-7c4b4701b779-kube-api-access-dk5nx\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.514577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.514419 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126a1b2-1b19-441d-912d-7c4b4701b779-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.616074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.616015 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5nx\" (UniqueName: \"kubernetes.io/projected/3126a1b2-1b19-441d-912d-7c4b4701b779-kube-api-access-dk5nx\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.616307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.616190 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126a1b2-1b19-441d-912d-7c4b4701b779-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.616901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.616875 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126a1b2-1b19-441d-912d-7c4b4701b779-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.630541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.630463 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5nx\" (UniqueName: \"kubernetes.io/projected/3126a1b2-1b19-441d-912d-7c4b4701b779-kube-api-access-dk5nx\") pod \"payload-pre-processing-7b98bb678c-h4h49\" (UID: \"3126a1b2-1b19-441d-912d-7c4b4701b779\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.680303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.680266 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:47.964415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.964300 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" podStartSLOduration=3.6242551130000003 podStartE2EDuration="6.964276913s" totalImagesPullingTime="3.3400218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:47.413692002 +0000 UTC m=+668.138699599" watchObservedRunningTime="2026-11-03 20:08:47.964276913 +0000 UTC m=+668.689284522" Nov 03 20:08:47.966719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:47.966690 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h4h49"] Nov 03 20:08:47.968314 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:47.968291 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3126a1b2_1b19_441d_912d_7c4b4701b779.slice/crio-2c48ad09ab70265cdf2117599fe5fb1c4e6944d722fc950e247de26c7eb9317a WatchSource:0}: Error finding container 2c48ad09ab70265cdf2117599fe5fb1c4e6944d722fc950e247de26c7eb9317a: Status 404 returned error can't find the container with id 2c48ad09ab70265cdf2117599fe5fb1c4e6944d722fc950e247de26c7eb9317a Nov 03 20:08:48.390481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:48.390440 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" event={"ID":"3126a1b2-1b19-441d-912d-7c4b4701b779","Type":"ContainerStarted","Data":"2c48ad09ab70265cdf2117599fe5fb1c4e6944d722fc950e247de26c7eb9317a"} Nov 03 20:08:49.254217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:49.254170 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:49.254401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:49.254237 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:50.295310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:50.295267 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:50.295796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:50.295346 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:50.403253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:50.403216 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" event={"ID":"3126a1b2-1b19-441d-912d-7c4b4701b779","Type":"ContainerStarted","Data":"53c255a7a6436d698ccde0e844a9a4b733bea58325c05a4e9c6a5dafb82e18f2"} Nov 03 20:08:50.403444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:50.403297 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:08:50.423448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:50.423327 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" podStartSLOduration=1.231301005 podStartE2EDuration="3.423309375s" totalImagesPullingTime="2.19200837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:50.422186603 +0000 UTC m=+671.147194200" watchObservedRunningTime="2026-11-03 20:08:50.423309375 +0000 UTC m=+671.148316972" Nov 03 20:08:51.336340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:51.336283 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:51.336873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:51.336352 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.439166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:51.439126 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:51.439573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:51.439549 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:51.440547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:51.440525 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:08:54.948850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:54.948817 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6"] Nov 03 20:08:54.952375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:54.952358 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:54.967348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:54.967322 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6"] Nov 03 20:08:55.098727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098694 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.098727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098726 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tv5c\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-kube-api-access-2tv5c\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.098942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098757 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.098942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098854 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.098942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098890 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.098942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.098928 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142de3d-195d-40ca-92b6-6777196e15d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.099092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.099007 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.099092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.099054 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.099092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.099069 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142de3d-195d-40ca-92b6-6777196e15d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200429 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200489 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142de3d-195d-40ca-92b6-6777196e15d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200543 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200586 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200604 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142de3d-195d-40ca-92b6-6777196e15d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200657 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200678 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tv5c\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-kube-api-access-2tv5c\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200717 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200756 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200897 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.200981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.200921 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.201239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.201148 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.201239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.201224 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.201417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.201397 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9142de3d-195d-40ca-92b6-6777196e15d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.203473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.203448 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9142de3d-195d-40ca-92b6-6777196e15d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.203723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.203708 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9142de3d-195d-40ca-92b6-6777196e15d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.213051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.213025 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.213318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.213301 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tv5c\" (UniqueName: \"kubernetes.io/projected/9142de3d-195d-40ca-92b6-6777196e15d5-kube-api-access-2tv5c\") pod \"maas-default-gateway-openshift-default-77f556d647-v6ll6\" (UID: \"9142de3d-195d-40ca-92b6-6777196e15d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.265958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.265933 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:55.547777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.547750 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6"] Nov 03 20:08:55.550940 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:08:55.550900 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9142de3d_195d_40ca_92b6_6777196e15d5.slice/crio-3e979b4572dbd97885e0a632b0d2282b9f9fb5800438cd5d9cae8d7a83a35680 WatchSource:0}: Error finding container 3e979b4572dbd97885e0a632b0d2282b9f9fb5800438cd5d9cae8d7a83a35680: Status 404 returned error can't find the container with id 3e979b4572dbd97885e0a632b0d2282b9f9fb5800438cd5d9cae8d7a83a35680 Nov 03 20:08:55.553203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.553170 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:55.553298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.553236 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:55.553298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:55.553266 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:56.444941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:56.444900 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" event={"ID":"9142de3d-195d-40ca-92b6-6777196e15d5","Type":"ContainerStarted","Data":"b2ea759fd582478cb980b9403b2e336f76b157c5820556e0c2e5423f1f7d50fb"} Nov 03 20:08:56.445384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:56.444997 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" event={"ID":"9142de3d-195d-40ca-92b6-6777196e15d5","Type":"ContainerStarted","Data":"3e979b4572dbd97885e0a632b0d2282b9f9fb5800438cd5d9cae8d7a83a35680"} Nov 03 20:08:56.471466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:56.471409 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podStartSLOduration=2.471396852 podStartE2EDuration="2.471396852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:56.470022898 +0000 UTC m=+677.195030507" watchObservedRunningTime="2026-11-03 20:08:56.471396852 +0000 UTC m=+677.196404448" Nov 03 20:08:57.266977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:57.266944 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:08:57.268596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:57.268573 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:57.268660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:57.268626 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:08:58.267013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.266980 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:58.267462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.267035 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:08:58.393077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.393049 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:08:58.650265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.650233 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7878f95ccc-l27rs"] Nov 03 20:08:58.654030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.654006 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.656504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.656481 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbxn\" (UniqueName: \"kubernetes.io/projected/a6709e2e-e552-4789-ad94-0788a9583418-kube-api-access-bkbxn\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.656654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.656629 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6709e2e-e552-4789-ad94-0788a9583418-metrics-certs\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.663760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.663733 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7878f95ccc-l27rs"] Nov 03 20:08:58.757364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.757328 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbxn\" (UniqueName: \"kubernetes.io/projected/a6709e2e-e552-4789-ad94-0788a9583418-kube-api-access-bkbxn\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.757542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.757428 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6709e2e-e552-4789-ad94-0788a9583418-metrics-certs\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.759798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.759771 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6709e2e-e552-4789-ad94-0788a9583418-metrics-certs\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.770082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.770056 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbxn\" (UniqueName: \"kubernetes.io/projected/a6709e2e-e552-4789-ad94-0788a9583418-kube-api-access-bkbxn\") pod \"ai-gateway-controller-7878f95ccc-l27rs\" (UID: \"a6709e2e-e552-4789-ad94-0788a9583418\") " pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:58.969972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:58.969884 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:08:59.267261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:59.267184 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:08:59.267261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:59.267243 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:08:59.275578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:59.275552 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7878f95ccc-l27rs"] Nov 03 20:08:59.461485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:08:59.461449 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" event={"ID":"a6709e2e-e552-4789-ad94-0788a9583418","Type":"ContainerStarted","Data":"702ec5ba5d27224894d855038863e3f6b0f58d7828204ef0d0b79e8f8b7f504b"} Nov 03 20:09:00.263920 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:09:00.263888 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:09:00.266285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:00.266264 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:09:00.266345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:00.266327 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:09:00.467738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:00.467695 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" event={"ID":"a6709e2e-e552-4789-ad94-0788a9583418","Type":"ContainerStarted","Data":"87e1159aca2ad7cde53fd4955ffd3c2194c38bac223e964b9e4002f4f01415b6"} Nov 03 20:09:00.468137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:00.467784 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:09:00.488470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:00.488427 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" podStartSLOduration=2.0527602 podStartE2EDuration="2.488414064s" totalImagesPullingTime="435.653864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:00.486951735 +0000 UTC m=+681.211959334" watchObservedRunningTime="2026-11-03 20:09:00.488414064 +0000 UTC m=+681.213421660" Nov 03 20:09:01.266775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:01.266738 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:09:01.266981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:01.266800 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:09:01.423670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:01.423645 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h4h49" Nov 03 20:09:02.266305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:02.266271 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:09:02.266823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:02.266332 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 20:09:03.746799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:03.746767 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:09:03.747334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:03.747125 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:09:03.748166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:03.748145 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" Nov 03 20:09:03.946783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:03.946753 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:11.475879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.475850 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7878f95ccc-l27rs" Nov 03 20:09:11.522159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.522130 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:09:11.522385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.522358 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" podUID="4b20221b-d7d0-4515-847d-02579b0c867a" containerName="manager" containerID="cri-o://2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d" gracePeriod=30 Nov 03 20:09:11.774191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.774165 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:09:11.892251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.892217 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") pod \"4b20221b-d7d0-4515-847d-02579b0c867a\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " Nov 03 20:09:11.892441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.892269 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\" (UniqueName: \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\") pod \"4b20221b-d7d0-4515-847d-02579b0c867a\" (UID: \"4b20221b-d7d0-4515-847d-02579b0c867a\") " Nov 03 20:09:11.894395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.894360 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs" pod "4b20221b-d7d0-4515-847d-02579b0c867a" (UID: "4b20221b-d7d0-4515-847d-02579b0c867a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:11.894506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.894431 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf" pod "4b20221b-d7d0-4515-847d-02579b0c867a" (UID: "4b20221b-d7d0-4515-847d-02579b0c867a"). InnerVolumeSpecName "kube-api-access-6pkvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:11.993865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.993832 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b20221b-d7d0-4515-847d-02579b0c867a-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:11.993865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:11.993859 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkvf\" (UniqueName: \"kubernetes.io/projected/4b20221b-d7d0-4515-847d-02579b0c867a-kube-api-access-6pkvf\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:12.535169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.535131 5954 generic.go:356] "Generic (PLEG): container finished" podID="4b20221b-d7d0-4515-847d-02579b0c867a" containerID="2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d" exitCode=0 Nov 03 20:09:12.535591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.535186 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" Nov 03 20:09:12.535591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.535215 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" event={"ID":"4b20221b-d7d0-4515-847d-02579b0c867a","Type":"ContainerDied","Data":"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d"} Nov 03 20:09:12.535591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.535270 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86c7698f44-clnb5" event={"ID":"4b20221b-d7d0-4515-847d-02579b0c867a","Type":"ContainerDied","Data":"fa17699d0b3153597f36f5399140815a540cd31227df1cc98f61a010a065a316"} Nov 03 20:09:12.535591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.535292 5954 scope.go:122] "RemoveContainer" containerID="2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d" Nov 03 20:09:12.545461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.545443 5954 scope.go:122] "RemoveContainer" containerID="2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d" Nov 03 20:09:12.545725 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:09:12.545708 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d\": container with ID starting with 2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d not found: ID does not exist" containerID="2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d" Nov 03 20:09:12.545766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.545733 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d"} err="failed to get container status \"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d\": rpc error: code = NotFound desc = could not find container \"2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d\": container with ID starting with 2dca31b9bc20aeeecec63da3891141c070d94e4776a6ed0a3fa1e4b5ffd77b2d not found: ID does not exist" Nov 03 20:09:12.557827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.557801 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:09:12.561622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:12.561597 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-86c7698f44-clnb5"] Nov 03 20:09:13.173005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:13.172968 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:13.404078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:13.404045 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b20221b-d7d0-4515-847d-02579b0c867a" path="/var/lib/kubelet/pods/4b20221b-d7d0-4515-847d-02579b0c867a/volumes" Nov 03 20:09:13.694431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:13.694394 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:15.316678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:15.316641 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:15.425428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:15.425392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:26.860283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:26.860246 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:26.959071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:26.959035 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:30.256863 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:09:30.256827 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:09:46.250795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:46.250746 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:46.345544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:46.345486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:47.159525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:47.159482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:47.898809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:47.898758 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" podUID="d161bc542803fe55ee71704a451b4d4d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:09:59.974489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:09:59.974460 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.073175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.073142 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.258168 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:10:00.258081 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:10:00.385501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.385465 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:10:00.385945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.385923 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4b20221b-d7d0-4515-847d-02579b0c867a" containerName="manager" Nov 03 20:10:00.385945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.385944 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b20221b-d7d0-4515-847d-02579b0c867a" containerName="manager" Nov 03 20:10:00.386139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.386056 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b20221b-d7d0-4515-847d-02579b0c867a" containerName="manager" Nov 03 20:10:00.389051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.389034 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.395819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.395753 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d412cf-openshift-default-dockercfg-qrnd5\"" Nov 03 20:10:00.402431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.402397 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:10:00.440516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.440486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.487069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487035 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwt7n\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487077 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487132 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487305 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487359 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487420 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487465 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487492 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.487582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.487508 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.548007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.547926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.571586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.571551 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:10:00.575740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.575300 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.589338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589313 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589357 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:10:00.589478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589373 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589401 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589451 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589486 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589505 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwt7n\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589594 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.589629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.589615 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.591097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.590196 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.591097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.590464 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.591097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.590705 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.591097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.590820 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.591097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.591066 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.593057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.593030 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.593287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.593268 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.595650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.595627 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8013d9-openshift-default-dockercfg-f7g58\"" Nov 03 20:10:00.621733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.621695 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.621904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.621867 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwt7n\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.665687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.665648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.690356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690318 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690443 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690603 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690668 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690771 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690805 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6xj\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690827 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.690982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.690958 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.706015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.705991 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:00.787783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.784154 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.792168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792134 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792211 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792256 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792317 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792372 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792404 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6xj\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792435 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792490 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.792561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.792546 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.794037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.793583 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.795250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.795224 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.795766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.795748 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.795899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.795874 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.796033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.796008 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.796465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.796257 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.797833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.797814 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.807446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.807395 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6xj\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.807748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.807724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.889833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.889801 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:00.894843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.894818 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.996059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.996023 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:00.997156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:00.997133 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:10:00.999512 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:00.999478 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fadff0_23ad_4fb0_8d19_4da6b7ec6957.slice/crio-9c860dfe60696c05ba1f5e1b1fe943f076b573aba35a6e1788d0ebb9f30f8579 WatchSource:0}: Error finding container 9c860dfe60696c05ba1f5e1b1fe943f076b573aba35a6e1788d0ebb9f30f8579: Status 404 returned error can't find the container with id 9c860dfe60696c05ba1f5e1b1fe943f076b573aba35a6e1788d0ebb9f30f8579 Nov 03 20:10:01.001610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.001582 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.001661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.001643 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.001696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.001676 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.172063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.171918 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:10:01.175039 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:01.175011 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b7fecc_5c4d_44be_adb0_30654c997e70.slice/crio-edd04eca3eaa6614083ddcade7c0b403de1ba57ccd59fe062a33514c80d3e916 WatchSource:0}: Error finding container edd04eca3eaa6614083ddcade7c0b403de1ba57ccd59fe062a33514c80d3e916: Status 404 returned error can't find the container with id edd04eca3eaa6614083ddcade7c0b403de1ba57ccd59fe062a33514c80d3e916 Nov 03 20:10:01.178035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.178005 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.178154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.178077 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.178231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.178155 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.792619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.792505 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerStarted","Data":"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc"} Nov 03 20:10:01.792619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.792581 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerStarted","Data":"edd04eca3eaa6614083ddcade7c0b403de1ba57ccd59fe062a33514c80d3e916"} Nov 03 20:10:01.796299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.795742 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerStarted","Data":"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10"} Nov 03 20:10:01.796299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.795817 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerStarted","Data":"9c860dfe60696c05ba1f5e1b1fe943f076b573aba35a6e1788d0ebb9f30f8579"} Nov 03 20:10:01.819018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.818963 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podStartSLOduration=1.818950098 podStartE2EDuration="1.818950098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.817182039 +0000 UTC m=+742.542189647" watchObservedRunningTime="2026-11-03 20:10:01.818950098 +0000 UTC m=+742.543958031" Nov 03 20:10:01.845890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.845842 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podStartSLOduration=1.84583006 podStartE2EDuration="1.84583006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.84479462 +0000 UTC m=+742.569802217" watchObservedRunningTime="2026-11-03 20:10:01.84583006 +0000 UTC m=+742.570837655" Nov 03 20:10:01.890561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:01.890527 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:02.706183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:02.706145 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:02.711609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:02.711583 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:02.800582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:02.800543 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:02.801329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:02.801312 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:02.896822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:02.896791 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:03.805459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:03.805427 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:03.806331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:03.806309 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:03.929543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:03.929508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.037036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.037002 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.245546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.245511 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.356608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.356571 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.456699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.456658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.573535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.573463 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:04.685514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:04.685486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:05.212477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.212445 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:05.322221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.322192 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:05.330660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.330632 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:10:05.334465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.334445 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.344128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.344092 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:10:05.414241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.410645 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a6f98b-dockercfg-nqgf7\"" Nov 03 20:10:05.414478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.411331 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a6f98b\"" Nov 03 20:10:05.448046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.448009 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn6j\" (UniqueName: \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.448939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.448913 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.550395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.550313 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn6j\" (UniqueName: \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.550537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.550419 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.551100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.551079 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.562318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.562296 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn6j\" (UniqueName: \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\") pod \"payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.651345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.651317 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:05.931986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:05.931949 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:10:05.932885 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:05.932855 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf041dbc_872e_4479_9b4d_12f3c153dce0.slice/crio-a6c23f00c0858dcb4408ecfc1d65e8aef872244a5e9b4f5efeb769bdbe6ab950 WatchSource:0}: Error finding container a6c23f00c0858dcb4408ecfc1d65e8aef872244a5e9b4f5efeb769bdbe6ab950: Status 404 returned error can't find the container with id a6c23f00c0858dcb4408ecfc1d65e8aef872244a5e9b4f5efeb769bdbe6ab950 Nov 03 20:10:06.000971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.000943 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:06.094676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.094641 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:10:06.098522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.098506 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.103797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.103774 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:06.106289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.106267 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-90ef9c\"" Nov 03 20:10:06.106802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.106782 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-90ef9c-dockercfg-l79bj\"" Nov 03 20:10:06.109748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.109726 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:10:06.168155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.168101 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:10:06.171979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.171958 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.184195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.184164 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:10:06.249951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.249922 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:06.258735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.258701 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthz6\" (UniqueName: \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.258865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.258765 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.364833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.364794 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthz6\" (UniqueName: \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.365498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.365471 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.365784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.365750 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.365900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.365823 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pqs\" (UniqueName: \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.366869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.366848 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.376938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.376918 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthz6\" (UniqueName: \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\") pod \"payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.413662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.413635 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.467031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.466930 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.467031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.467005 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pqs\" (UniqueName: \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.467968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.467945 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.479643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.479617 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pqs\" (UniqueName: \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\") pod \"payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.490896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.490652 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:06.648661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.648622 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:06.705952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.705930 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:10:06.708003 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:06.707971 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de87552_80de_44bc_9487_058fcd3cbabe.slice/crio-f557f52600cce58d605732f720a833a34f6199d0e2f490306b0a9292a6207b6f WatchSource:0}: Error finding container f557f52600cce58d605732f720a833a34f6199d0e2f490306b0a9292a6207b6f: Status 404 returned error can't find the container with id f557f52600cce58d605732f720a833a34f6199d0e2f490306b0a9292a6207b6f Nov 03 20:10:06.759832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.759798 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:06.807319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.807297 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:10:06.809240 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:06.809216 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5b16f2_bfeb_4339_9660_32090c2e89ab.slice/crio-195d4d3c2d7f395fd05e766d961531e61fd94af94d2f7a4cbba10192ed16b92e WatchSource:0}: Error finding container 195d4d3c2d7f395fd05e766d961531e61fd94af94d2f7a4cbba10192ed16b92e: Status 404 returned error can't find the container with id 195d4d3c2d7f395fd05e766d961531e61fd94af94d2f7a4cbba10192ed16b92e Nov 03 20:10:06.834872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.834839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" event={"ID":"9de87552-80de-44bc-9487-058fcd3cbabe","Type":"ContainerStarted","Data":"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5"} Nov 03 20:10:06.836937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.834907 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" event={"ID":"9de87552-80de-44bc-9487-058fcd3cbabe","Type":"ContainerStarted","Data":"f557f52600cce58d605732f720a833a34f6199d0e2f490306b0a9292a6207b6f"} Nov 03 20:10:06.836937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.835882 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:06.865742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.864182 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" event={"ID":"df041dbc-872e-4479-9b4d-12f3c153dce0","Type":"ContainerStarted","Data":"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f"} Nov 03 20:10:06.865742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.864250 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" event={"ID":"df041dbc-872e-4479-9b4d-12f3c153dce0","Type":"ContainerStarted","Data":"a6c23f00c0858dcb4408ecfc1d65e8aef872244a5e9b4f5efeb769bdbe6ab950"} Nov 03 20:10:06.865742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.864443 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:06.866083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.866042 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" event={"ID":"0d5b16f2-bfeb-4339-9660-32090c2e89ab","Type":"ContainerStarted","Data":"195d4d3c2d7f395fd05e766d961531e61fd94af94d2f7a4cbba10192ed16b92e"} Nov 03 20:10:06.932460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:06.932410 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" podStartSLOduration=0.932395314 podStartE2EDuration="932.395314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:06.888159505 +0000 UTC m=+747.613167100" watchObservedRunningTime="2026-11-03 20:10:06.932395314 +0000 UTC m=+747.657402932" Nov 03 20:10:07.047256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.047166 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:07.366398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.366367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:07.416302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.416247 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" podStartSLOduration=2.416233867 podStartE2EDuration="2.416233867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:06.932288439 +0000 UTC m=+747.657296035" watchObservedRunningTime="2026-11-03 20:10:07.416233867 +0000 UTC m=+748.141241461" Nov 03 20:10:07.416455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.416370 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:10:07.419962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.419942 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.427158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.427134 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8013d9\"" Nov 03 20:10:07.429706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.429687 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:10:07.479928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.479899 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.480071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.479986 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.488527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.488499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:07.587437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.580780 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.587437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.580884 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.587437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.581764 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.599884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.599848 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\") pod \"payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.734865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.734793 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:07.889202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.889160 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" event={"ID":"0d5b16f2-bfeb-4339-9660-32090c2e89ab","Type":"ContainerStarted","Data":"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1"} Nov 03 20:10:07.889475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.889457 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:07.909019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:07.908964 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" podStartSLOduration=1.908951694 podStartE2EDuration="1.908951694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:07.906626288 +0000 UTC m=+748.631633885" watchObservedRunningTime="2026-11-03 20:10:07.908951694 +0000 UTC m=+748.633959289" Nov 03 20:10:08.013359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.013324 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:10:08.015579 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:08.015554 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcfa068_2541_44f6_927c_bc25b97659ed.slice/crio-b7712cd03ea62b41c8a76bc585f5e1c9131017b7b7fefd2105df4ea7ef233180 WatchSource:0}: Error finding container b7712cd03ea62b41c8a76bc585f5e1c9131017b7b7fefd2105df4ea7ef233180: Status 404 returned error can't find the container with id b7712cd03ea62b41c8a76bc585f5e1c9131017b7b7fefd2105df4ea7ef233180 Nov 03 20:10:08.251145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.251086 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:08.315294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.315223 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:10:08.318973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.318953 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.327358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.327331 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-d412cf-dockercfg-j66kp\"" Nov 03 20:10:08.327494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.327479 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-d412cf\"" Nov 03 20:10:08.329999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.329979 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:10:08.354396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.354364 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:10:08.355534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.355505 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:08.358357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.358337 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.368057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.368026 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:10:08.388926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.388893 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.389070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.388960 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlrc\" (UniqueName: \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.389187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.389170 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkhl\" (UniqueName: \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.389232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.389206 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.477800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.477745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:08.490639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.490614 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkhl\" (UniqueName: \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.490780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.490649 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.490848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.490801 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.490903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.490847 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlrc\" (UniqueName: \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.491574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.491543 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.491700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.491685 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.502778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.502746 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkhl\" (UniqueName: \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\") pod \"payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.502859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.502812 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlrc\" (UniqueName: \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\") pod \"payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.632121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.632077 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:08.648046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.648016 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:08.674252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.674223 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:08.898235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.898140 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" event={"ID":"2fcfa068-2541-44f6-927c-bc25b97659ed","Type":"ContainerStarted","Data":"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d"} Nov 03 20:10:08.898235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.898210 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" event={"ID":"2fcfa068-2541-44f6-927c-bc25b97659ed","Type":"ContainerStarted","Data":"b7712cd03ea62b41c8a76bc585f5e1c9131017b7b7fefd2105df4ea7ef233180"} Nov 03 20:10:08.898456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.898438 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:08.920340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.920284 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:10:08.933199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.926165 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" podStartSLOduration=1.926147785 podStartE2EDuration="1.926147785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:08.925609535 +0000 UTC m=+749.650617128" watchObservedRunningTime="2026-11-03 20:10:08.926147785 +0000 UTC m=+749.651155381" Nov 03 20:10:08.960155 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:08.957427 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa553fb_c462_4556_8684_4e78da36aa00.slice/crio-b2792fa3add155d711885506690a58e742333653c0fa17fe00126c16869e6d5d WatchSource:0}: Error finding container b2792fa3add155d711885506690a58e742333653c0fa17fe00126c16869e6d5d: Status 404 returned error can't find the container with id b2792fa3add155d711885506690a58e742333653c0fa17fe00126c16869e6d5d Nov 03 20:10:08.989877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:08.989855 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:10:08.992206 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:08.992179 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddd9bf4_22f0_400b_957a_7a090243bdd0.slice/crio-3ac0f319a81495a23051ca36a6ff00f772f4fa66816237ce18edc61195200f1f WatchSource:0}: Error finding container 3ac0f319a81495a23051ca36a6ff00f772f4fa66816237ce18edc61195200f1f: Status 404 returned error can't find the container with id 3ac0f319a81495a23051ca36a6ff00f772f4fa66816237ce18edc61195200f1f Nov 03 20:10:09.283588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.283496 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:09.385190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.384536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:09.541541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.541458 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:09.907527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.907478 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" event={"ID":"aaa553fb-c462-4556-8684-4e78da36aa00","Type":"ContainerStarted","Data":"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f"} Nov 03 20:10:09.907705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.907544 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" event={"ID":"aaa553fb-c462-4556-8684-4e78da36aa00","Type":"ContainerStarted","Data":"b2792fa3add155d711885506690a58e742333653c0fa17fe00126c16869e6d5d"} Nov 03 20:10:09.907780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.907722 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:09.909369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.909339 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" event={"ID":"addd9bf4-22f0-400b-957a-7a090243bdd0","Type":"ContainerStarted","Data":"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04"} Nov 03 20:10:09.909500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.909389 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" event={"ID":"addd9bf4-22f0-400b-957a-7a090243bdd0","Type":"ContainerStarted","Data":"3ac0f319a81495a23051ca36a6ff00f772f4fa66816237ce18edc61195200f1f"} Nov 03 20:10:09.909500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.909439 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:09.926575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.926529 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" podStartSLOduration=1.926516628 podStartE2EDuration="1.926516628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:09.925784216 +0000 UTC m=+750.650791812" watchObservedRunningTime="2026-11-03 20:10:09.926516628 +0000 UTC m=+750.651524217" Nov 03 20:10:09.945774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.945728 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" podStartSLOduration=1.9457164329999999 podStartE2EDuration="1.945716433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:09.944546679 +0000 UTC m=+750.669554298" watchObservedRunningTime="2026-11-03 20:10:09.945716433 +0000 UTC m=+750.670724054" Nov 03 20:10:09.964031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:09.963992 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.197916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.197838 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.365150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.365118 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:10:10.369216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.369193 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.378462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.378439 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-62221d-openshift-default-dockercfg-tqhkb\"" Nov 03 20:10:10.385000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.384977 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:10:10.415024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.414997 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415034 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415061 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6ws\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415244 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415302 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415393 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415424 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.415560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.415539 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.455991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.455891 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.518782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.518739 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.518952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.518808 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6ws\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.518952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.518872 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.518952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.518916 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.518981 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.519013 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.519063 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.519089 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.519215 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.519653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.519622 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.527266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.523729 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.527266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.523884 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.527266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.524122 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.527266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.524353 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.531309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.531286 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.533689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.533670 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.533923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.533900 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6ws\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.542989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.542964 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.684313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.684280 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:10.691347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.691316 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.980850 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:10.980814 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f3785c_6715_4731_99f7_3b9f38c90fa3.slice/crio-a977daf411e0c093bbdd7be84f97edf697784f0c8878120ff26681db20cdffe1 WatchSource:0}: Error finding container a977daf411e0c093bbdd7be84f97edf697784f0c8878120ff26681db20cdffe1: Status 404 returned error can't find the container with id a977daf411e0c093bbdd7be84f97edf697784f0c8878120ff26681db20cdffe1 Nov 03 20:10:10.981133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.981086 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:10:10.983306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.983277 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:10.983408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.983334 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:10.983408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:10.983362 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:11.121210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.121177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:11.223436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.223404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:11.911141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.911093 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:11.920865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.920838 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerStarted","Data":"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42"} Nov 03 20:10:11.921004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.920896 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerStarted","Data":"a977daf411e0c093bbdd7be84f97edf697784f0c8878120ff26681db20cdffe1"} Nov 03 20:10:11.948228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:11.948187 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podStartSLOduration=1.948174334 podStartE2EDuration="1.948174334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:11.945923868 +0000 UTC m=+752.670931486" watchObservedRunningTime="2026-11-03 20:10:11.948174334 +0000 UTC m=+752.673182000" Nov 03 20:10:12.011409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.011373 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:12.115910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.112051 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:12.684639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.684599 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:12.689740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.689713 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:12.723356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.723319 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:12.926864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.926837 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:12.927680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.927664 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:12.939389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:12.939325 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:13.975926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:13.975895 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:14.084422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:14.084390 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:14.189816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:14.189779 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:14.345401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:14.345315 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:14.444525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:14.444494 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:15.252969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:15.252931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:15.350689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:15.350656 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:15.454756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:15.454723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:16.075781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.075745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:16.142152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.142122 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:10:16.147137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.147078 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:10:16.147276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.147225 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.158035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.158009 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-62221d\"" Nov 03 20:10:16.159335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.159315 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-62221d-dockercfg-hndkc\"" Nov 03 20:10:16.183746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.183716 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:16.206199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.206171 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.206865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.206394 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmxn\" (UniqueName: \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.349300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.326806 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.351445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.351406 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmxn\" (UniqueName: \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.351820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.351790 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.363512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.363487 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmxn\" (UniqueName: \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\") pod \"payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.467186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.467152 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.583291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.583257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:16.767506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.767473 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:10:16.771369 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:16.771339 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9c0e0b_7d3d_4950_9175_2e5b2ff7d7ff.slice/crio-ff0fe0da233475791193ace6b8b138ec221627c3dac5ad5c56adf0b3ac8d8d65 WatchSource:0}: Error finding container ff0fe0da233475791193ace6b8b138ec221627c3dac5ad5c56adf0b3ac8d8d65: Status 404 returned error can't find the container with id ff0fe0da233475791193ace6b8b138ec221627c3dac5ad5c56adf0b3ac8d8d65 Nov 03 20:10:16.958802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.958767 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" event={"ID":"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff","Type":"ContainerStarted","Data":"1658dfc4fa260bf3c7db7357279800e6afb6b706ea5e2f738c9afd88dc09b86b"} Nov 03 20:10:16.958964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.958821 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" event={"ID":"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff","Type":"ContainerStarted","Data":"ff0fe0da233475791193ace6b8b138ec221627c3dac5ad5c56adf0b3ac8d8d65"} Nov 03 20:10:16.958964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.958850 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:16.978894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:16.978842 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" podStartSLOduration=0.978830222 podStartE2EDuration="978.830222ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:16.97644679 +0000 UTC m=+757.701454395" watchObservedRunningTime="2026-11-03 20:10:16.978830222 +0000 UTC m=+757.703837817" Nov 03 20:10:17.567695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:17.567663 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:17.714145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:17.714083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:17.892365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:17.892338 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:10:18.891995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:18.891961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:18.901710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:18.901689 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:10:18.901844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:18.901751 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:10:19.096745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:19.096715 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:19.530132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:19.521606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:19.633577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:19.633504 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:19.758389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:19.758265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:19.937131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:19.933286 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:10:20.245691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:20.245589 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:20.937130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:20.927624 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:10:20.937130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:20.931312 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:10:21.380272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:21.380237 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:21.479901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:21.479867 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:21.606648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:21.606616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:22.150781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.150745 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:10:22.156856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.156830 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.165828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.165800 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:10:22.179371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.179267 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-62221d-serving-cert\"" Nov 03 20:10:22.179652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.179631 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:10:22.179862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.179821 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.190008 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wdn\" (UniqueName: \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.190175 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.191304 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.191413 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.192394 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:10:22.196152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.192498 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:10:22.199146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.197373 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:22.290545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.290510 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:22.292798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.292776 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wdn\" (UniqueName: \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.292886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.292866 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.292950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.292893 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.292950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.292936 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.293055 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:10:22.293041 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-serving-cert: secret "maas-api-e2e-worker-w3-62221d-serving-cert" not found Nov 03 20:10:22.293128 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:10:22.293123 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls podName:0220a2f9-47ea-4a5a-aed5-66a166d2b0fe nodeName:}" failed. No retries permitted until 2026-11-03 20:10:22.793093362 +0000 UTC m=+763.518100940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls") pod "maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" (UID: "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe") : secret "maas-api-e2e-worker-w3-62221d-serving-cert" not found Nov 03 20:10:22.293600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.293582 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.295411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.295387 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.306165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.306140 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wdn\" (UniqueName: \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.631084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.630935 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:22.805467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.805397 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:22.810601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:22.810569 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") pod \"maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:23.087141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:23.087049 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:23.111464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:23.111430 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:23.439149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:23.427621 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:10:23.445045 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:10:23.445014 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0220a2f9_47ea_4a5a_aed5_66a166d2b0fe.slice/crio-b311615d1f74cb06671976c0573303a71d31c60de1095c48e54aa1f657f01b0f WatchSource:0}: Error finding container b311615d1f74cb06671976c0573303a71d31c60de1095c48e54aa1f657f01b0f: Status 404 returned error can't find the container with id b311615d1f74cb06671976c0573303a71d31c60de1095c48e54aa1f657f01b0f Nov 03 20:10:23.446420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:23.446398 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:10:24.065436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.065335 5954 generic.go:356] "Generic (PLEG): container finished" podID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerID="8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc" exitCode=0 Nov 03 20:10:24.065436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.065418 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerDied","Data":"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc"} Nov 03 20:10:24.065886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.065863 5954 scope.go:122] "RemoveContainer" containerID="8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc" Nov 03 20:10:24.068288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.068262 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" event={"ID":"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe","Type":"ContainerStarted","Data":"b311615d1f74cb06671976c0573303a71d31c60de1095c48e54aa1f657f01b0f"} Nov 03 20:10:24.069060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.069030 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:24.069189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.069128 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:24.069189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.069183 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:24.118178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.116197 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:24.364190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.364126 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:24.975656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:24.975628 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:25.084787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.084715 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:25.102314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.102277 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerStarted","Data":"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5"} Nov 03 20:10:25.175168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.175135 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:25.240483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.240392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:25.409525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.409496 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:25.890860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.890071 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:25.890860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.890493 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:10:25.890860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:25.890559 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:10:26.873795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:26.872667 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:26.890237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:26.890206 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:10:26.890386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:26.890256 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:10:27.891337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:27.891302 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:10:27.891848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:27.891365 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:10:27.982035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:27.981200 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:10:28.005684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:28.005655 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:28.891155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:28.891122 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:10:28.891323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:28.891174 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:10:29.106202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.106173 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:29.218213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.213367 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" event={"ID":"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe","Type":"ContainerStarted","Data":"03a2f80ff96d52287f9c7548944dc48736532037674c963bd8dedf3a4bf096d1"} Nov 03 20:10:29.218213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.214207 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:29.252144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.236870 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" podStartSLOduration=2.292031581 podStartE2EDuration="7.236850218s" totalImagesPullingTime="4.944818637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:29.235309921 +0000 UTC m=+769.960317533" watchObservedRunningTime="2026-11-03 20:10:29.236850218 +0000 UTC m=+769.961857817" Nov 03 20:10:29.469208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.462618 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:29.564464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.564433 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:29.890698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.890662 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:10:29.890882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:29.890725 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:10:30.056180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.055471 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.189148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.185658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.264516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.264488 5954 generic.go:356] "Generic (PLEG): container finished" podID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerID="7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10" exitCode=0 Nov 03 20:10:30.264723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.264564 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerDied","Data":"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10"} Nov 03 20:10:30.265031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.265010 5954 scope.go:122] "RemoveContainer" containerID="7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10" Nov 03 20:10:30.266524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.266487 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:30.266600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.266578 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:30.266640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.266621 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:30.295984 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:10:30.295827 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:10:30.303581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.303546 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.446206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.440686 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:30.976693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.976655 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:30.978156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.978076 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:30.979900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:30.979880 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:10:31.273857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:31.273777 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerStarted","Data":"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441"} Nov 03 20:10:31.451312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:31.451271 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:31.707704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:31.707234 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:31.708584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:31.708557 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:10:31.708739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:31.708619 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:10:32.442130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:32.441885 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:32.444977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:32.444950 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:32.707216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:32.707132 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:10:32.707216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:32.707194 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:10:32.800781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:32.800730 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:33.306744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:33.306699 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:33.706510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:33.706480 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:10:33.706974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:33.706575 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:10:33.747058 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:33.747009 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:34.020870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.020589 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:34.177793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.177730 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:34.663735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.663669 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:34.706741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.706706 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:10:34.707162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.706773 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:10:34.772212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.772085 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:34.872051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:34.871977 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:35.032712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:35.032580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:35.707461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:35.707212 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 20:10:35.707461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:35.707287 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 20:10:35.800571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:35.800537 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:36.280272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:36.280217 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:10:36.302287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:36.302246 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:36.687455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:36.687376 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:36.983569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:36.983489 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:36.984821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:36.984799 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:10:37.313771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.313689 5954 generic.go:356] "Generic (PLEG): container finished" podID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerID="97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42" exitCode=0 Nov 03 20:10:37.313933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.313761 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerDied","Data":"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42"} Nov 03 20:10:37.314251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.314233 5954 scope.go:122] "RemoveContainer" containerID="97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42" Nov 03 20:10:37.315457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.315431 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:37.315538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.315491 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:37.315538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.315523 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:37.759230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:37.759193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:38.342850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.342809 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerStarted","Data":"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69"} Nov 03 20:10:38.599850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.599770 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:38.684958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.684891 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:38.686207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.686182 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:10:38.686312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.686239 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 20:10:38.808447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:38.808413 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:39.601323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:39.601267 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:39.684722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:39.684687 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:10:39.684917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:39.684743 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 20:10:40.684947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:40.684911 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:10:40.685366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:40.684974 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 20:10:40.787159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:40.787126 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:41.684807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:41.684772 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:10:41.684990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:41.684836 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 20:10:41.689858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:41.689804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:42.503764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:42.503724 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:42.685578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:42.685546 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:10:42.685946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:42.685595 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 20:10:42.728146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:42.728098 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:42.937130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:42.928280 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:43.393451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:43.393404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:43.689544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:43.689469 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:43.690586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:43.690568 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:10:44.191021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:44.190986 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:44.641511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:44.641481 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:45.087813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:45.087708 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:45.290535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:45.290467 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:45.399696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:45.399668 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:45.509004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:45.508900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:45.942154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:45.942122 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:46.725045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:46.725013 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:46.798911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:46.798881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:47.219618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:47.219584 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:47.798244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:47.798212 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:48.582424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:48.582389 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:49.194613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:49.194580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:49.286383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:49.286355 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:50.086373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:50.086337 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:50.786421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:50.786388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:51.488683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:51.488648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:51.807960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:51.807865 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:52.429913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:52.429882 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:53.301045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:53.301008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:54.089434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:54.089397 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:54.799739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:54.799709 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:55.502484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:55.502453 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:56.075554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:56.075522 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:10:56.197776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:56.197748 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:57.084787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:57.084752 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:57.881913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:57.881875 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:58.223998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:58.223901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:58.685205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:58.685167 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:59.483035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:10:59.482994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.182008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:00.181970 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.274562 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:11:00.274526 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:11:00.985731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:00.985695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:01.778772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:01.778685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:02.221622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:02.221590 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:02.786427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:02.786388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:03.487440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:03.487409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:03.679565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:03.679536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:04.220130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:04.220076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:04.880770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:04.880737 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:05.576688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.576657 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:05.896775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.896743 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:11:05.903627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.903606 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.909240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.909212 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"default-dockercfg-5lshz\"" Nov 03 20:11:05.909390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.909298 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"premium-w0-b1b190-kserve-self-signed-certs\"" Nov 03 20:11:05.911291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.911259 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:11:05.945754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.945723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:05.956035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956003 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.956181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956072 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.956238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956172 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.956283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956244 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.956283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956275 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.956398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.956380 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfgv\" (UniqueName: \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:05.973416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.973392 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"openshift-service-ca.crt\"" Nov 03 20:11:05.973416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:05.973392 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"kube-root-ca.crt\"" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.059703 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.059790 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.059837 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.059909 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.059952 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.060896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.060035 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfgv\" (UniqueName: \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.061739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.061715 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.062694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.062083 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.062840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.062614 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.067582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.067560 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.072311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.072252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.077794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.077762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfgv\" (UniqueName: \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\") pod \"premium-w0-b1b190-kserve-8c796b64f-jgg6n\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.218168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.218062 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:06.359691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.359654 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:06.495788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:06.495768 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:11:06.497887 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:11:06.497857 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4654d15_6a0d_4eb5_aae2_696f5209aa0c.slice/crio-cbb2dc7489c9f767eced3ee6f6e7cc17d48c3ea6d27ccc5204ae5e05999df06a WatchSource:0}: Error finding container cbb2dc7489c9f767eced3ee6f6e7cc17d48c3ea6d27ccc5204ae5e05999df06a: Status 404 returned error can't find the container with id cbb2dc7489c9f767eced3ee6f6e7cc17d48c3ea6d27ccc5204ae5e05999df06a Nov 03 20:11:07.290881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:07.290851 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:07.502923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:07.502839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" event={"ID":"c4654d15-6a0d-4eb5-aae2-696f5209aa0c","Type":"ContainerStarted","Data":"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8"} Nov 03 20:11:07.502923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:07.502884 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" event={"ID":"c4654d15-6a0d-4eb5-aae2-696f5209aa0c","Type":"ContainerStarted","Data":"cbb2dc7489c9f767eced3ee6f6e7cc17d48c3ea6d27ccc5204ae5e05999df06a"} Nov 03 20:11:07.525830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:07.525768 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podStartSLOduration=2.525750717 podStartE2EDuration="2.525750717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:07.521904035 +0000 UTC m=+808.246911630" watchObservedRunningTime="2026-11-03 20:11:07.525750717 +0000 UTC m=+808.250758312" Nov 03 20:11:08.094656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:08.094625 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:08.219023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:08.218990 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:08.807242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:08.807204 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:09.593064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:09.593030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:10.303000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:10.302961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:10.475945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:10.475909 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:11:11.190292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:11.190259 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:12.174563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:12.174532 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:12.978424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:12.978393 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:13.684979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:13.684936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:14.042928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:14.042833 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:14.597354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:14.597322 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:15.294124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:15.294067 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:15.992787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:15.992754 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:16.219083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:16.219047 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:16.231327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:16.231304 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:16.463666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:16.463638 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:16.599737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:16.599710 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:11:16.730555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:16.730483 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:17.407995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:17.407967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:18.104291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:18.104259 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:18.790510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:18.790480 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:19.618432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:19.618401 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:20.396096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:20.396007 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:21.070325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.070290 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:11:21.075612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.075589 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.085627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.085607 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"default-dockercfg-d2pzc\"" Nov 03 20:11:21.085863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.085646 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"openshift-service-ca.crt\"" Nov 03 20:11:21.085863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.085656 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"kube-root-ca.crt\"" Nov 03 20:11:21.086345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.086325 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:11:21.086527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.086512 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"premium-w3-62221d-kserve-self-signed-certs\"" Nov 03 20:11:21.102707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.102679 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.242725 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.242784 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.242880 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.242926 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptfl\" (UniqueName: \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.242963 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.243128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.243038 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.255808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.255775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:21.344303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344225 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344318 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344345 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344406 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344441 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptfl\" (UniqueName: \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344465 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344850 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.344983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.344890 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.346838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.346811 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.348043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.348022 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.359073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.359053 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptfl\" (UniqueName: \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\") pod \"premium-w3-62221d-kserve-5dd9745dd4-kk5ns\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.389072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.389050 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:21.702020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:21.701984 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:11:21.703778 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:11:21.703748 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0eb303_045b_47a0_a42a_4fd19eecc3ed.slice/crio-ec67e0add7fa62c8670471e18485adde54096200a54196b3a94be4e7c44b324b WatchSource:0}: Error finding container ec67e0add7fa62c8670471e18485adde54096200a54196b3a94be4e7c44b324b: Status 404 returned error can't find the container with id ec67e0add7fa62c8670471e18485adde54096200a54196b3a94be4e7c44b324b Nov 03 20:11:22.301982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:22.301923 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:22.614884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:22.614849 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" event={"ID":"7b0eb303-045b-47a0-a42a-4fd19eecc3ed","Type":"ContainerStarted","Data":"45fff91e862076de240ddb5f5256b7c9edd2821cac7c06ce6e5d2809fa0ab833"} Nov 03 20:11:22.615091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:22.614894 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" event={"ID":"7b0eb303-045b-47a0-a42a-4fd19eecc3ed","Type":"ContainerStarted","Data":"ec67e0add7fa62c8670471e18485adde54096200a54196b3a94be4e7c44b324b"} Nov 03 20:11:22.636379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:22.636331 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podStartSLOduration=1.636318406 podStartE2EDuration="1.636318406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:22.635580516 +0000 UTC m=+823.360588111" watchObservedRunningTime="2026-11-03 20:11:22.636318406 +0000 UTC m=+823.361325998" Nov 03 20:11:23.237896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:23.237860 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:23.389985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:23.389948 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:24.342518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:24.342484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:25.279466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:25.279431 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:25.996861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:25.996829 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:26.679528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:26.679501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:27.483295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:27.483261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:28.268969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:28.268937 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:29.165021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:29.164981 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.003394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.003365 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.257527 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:11:30.257449 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:11:30.659926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.659892 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:11:30.664338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.664317 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.678202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.678176 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3aba31-openshift-default-dockercfg-s4h77\"" Nov 03 20:11:30.681197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.681173 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:11:30.714638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.714609 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.770817 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.770867 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.770907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.770958 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.770986 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.771051 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.771168 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.771228 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbv5\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.772219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.771263 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.794212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.794179 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:11:30.798047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.798031 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.804728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.804698 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b1b190-openshift-default-dockercfg-g8zqz\"" Nov 03 20:11:30.818897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.818862 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:11:30.872786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.872755 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.872944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.872792 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.872944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.872854 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.872941 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.872996 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbv5\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873030 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873101 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873163 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873204 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873244 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873256 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873469 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873489 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.873806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.873785 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.875312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.875293 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.875438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.875422 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.885176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.885155 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.885326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.885308 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbv5\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:30.974278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974197 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974337 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974392 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974418 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974508 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77kd\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974545 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974574 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974622 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.974695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.974686 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:30.977831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:30.977814 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:31.075558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075519 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075627 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075682 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075816 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075872 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77kd\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075912 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075938 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.075976 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076033 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076192 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076223 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076476 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076687 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.076873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.076849 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.078019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.077997 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.078496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.078477 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.088131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.088082 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.088549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.088527 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77kd\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.111278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.111258 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:31.260023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.259994 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:11:31.261339 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:11:31.261313 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb994dc_361f_460a_8b12_6ad197b08849.slice/crio-35d18ea80fd27c5a1de369cffb2a1cf7c2540e542def9505479a040c5a6ca39f WatchSource:0}: Error finding container 35d18ea80fd27c5a1de369cffb2a1cf7c2540e542def9505479a040c5a6ca39f: Status 404 returned error can't find the container with id 35d18ea80fd27c5a1de369cffb2a1cf7c2540e542def9505479a040c5a6ca39f Nov 03 20:11:31.263684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.263658 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.263734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.263717 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.263766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.263745 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.389925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.389877 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:31.392718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.392693 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:11:31.394298 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:11:31.394268 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399211f6_d373_45c5_abc6_ff9d6b671d3d.slice/crio-aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3 WatchSource:0}: Error finding container aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3: Status 404 returned error can't find the container with id aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3 Nov 03 20:11:31.397132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.397085 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.397236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.397179 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.397236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.397220 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:31.407229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.407211 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:31.483885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.483825 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:31.667176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.667139 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" event={"ID":"aeb994dc-361f-460a-8b12-6ad197b08849","Type":"ContainerStarted","Data":"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7"} Nov 03 20:11:31.667375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.667202 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" event={"ID":"aeb994dc-361f-460a-8b12-6ad197b08849","Type":"ContainerStarted","Data":"35d18ea80fd27c5a1de369cffb2a1cf7c2540e542def9505479a040c5a6ca39f"} Nov 03 20:11:31.668886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.668857 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" event={"ID":"399211f6-d373-45c5-abc6-ff9d6b671d3d","Type":"ContainerStarted","Data":"a9589d5db2c486e02b9cb51c9bcb574a29a7f7a65dcdcf9a36d6af3a77d25da2"} Nov 03 20:11:31.669023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.668914 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" event={"ID":"399211f6-d373-45c5-abc6-ff9d6b671d3d","Type":"ContainerStarted","Data":"aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3"} Nov 03 20:11:31.682208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.682185 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:11:31.691667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.691624 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podStartSLOduration=1.6916107569999999 podStartE2EDuration="1.691610757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:31.690327588 +0000 UTC m=+832.415335207" watchObservedRunningTime="2026-11-03 20:11:31.691610757 +0000 UTC m=+832.416618352" Nov 03 20:11:31.978264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:31.978185 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:32.112343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.112306 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:32.113935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.113907 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:32.114190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.114163 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:32.295813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.295718 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:32.980206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.980174 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:11:32.980389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.980241 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:11:32.994787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:32.994753 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:33.112223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.112186 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:33.112396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.112260 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:33.697433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.697404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:33.860131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.859737 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podStartSLOduration=3.859714355 podStartE2EDuration="3.859714355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:31.744838337 +0000 UTC m=+832.469845932" watchObservedRunningTime="2026-11-03 20:11:33.859714355 +0000 UTC m=+834.584721966" Nov 03 20:11:33.865336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.863048 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:11:33.978737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.978655 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:11:33.978737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:33.978708 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:11:34.111676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:34.111634 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:34.111878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:34.111699 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:34.685655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:34.685600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:34.981187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:34.978555 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:11:34.981187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:34.978616 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:11:35.117131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:35.112376 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:35.117131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:35.112485 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:35.705131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:35.705020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:35.985189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:35.984513 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:11:35.985189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:35.984576 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:11:36.111765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:36.111728 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:36.111985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:36.111781 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:36.508154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:36.508123 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:36.978228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:36.978192 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:11:36.978647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:36.978273 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 20:11:37.112722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:37.112674 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:37.112920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:37.112755 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:37.363018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:37.362984 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:38.087607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.087570 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:38.111984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.111950 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:11:38.112167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.112016 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:11:38.169053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.167438 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:38.169053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.168892 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:38.173320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:38.173091 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:11:39.195738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:39.195707 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:39.196238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:39.196122 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:39.197253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:39.197234 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:11:39.292816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:39.292785 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:40.150619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:40.150577 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:40.885745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:40.885714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:41.578793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:41.578756 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:42.295404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:42.295362 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:43.005941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:43.005900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:43.697142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:43.697066 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:44.576610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:44.576560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:45.308322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:45.308270 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:45.916136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:45.916093 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:11:45.921785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:45.921758 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:45.929587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:45.929042 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90ef9c-openshift-default-dockercfg-9s9f5\"" Nov 03 20:11:45.932333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:45.932309 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:11:46.008659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.008609 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.008820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.008738 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.008820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.008796 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.008956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.008938 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.009066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.009042 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.009131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.009098 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.009178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.009166 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.009227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.009214 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.009269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.009244 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vb6\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.091038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.091007 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:46.110489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110453 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110584 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vb6\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110623 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110667 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110723 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110818 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110831 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.110994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.110938 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111000 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111026 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111038 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111259 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111318 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.111605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.111578 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.113045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.113024 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.113294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.113272 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.122693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.122671 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.122814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.122794 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vb6\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.237829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.237738 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:46.521709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.521683 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:11:46.523662 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:11:46.523633 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea8e0cd_ff84_484e_adad_c3552e6f1161.slice/crio-6da9ce9c524c010d5e94749de1bfdde513755a2aab0fa7a0caeba8dce6c3099c WatchSource:0}: Error finding container 6da9ce9c524c010d5e94749de1bfdde513755a2aab0fa7a0caeba8dce6c3099c: Status 404 returned error can't find the container with id 6da9ce9c524c010d5e94749de1bfdde513755a2aab0fa7a0caeba8dce6c3099c Nov 03 20:11:46.526161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.526099 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.526246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.526212 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.526283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.526252 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.808713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.808610 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" event={"ID":"2ea8e0cd-ff84-484e-adad-c3552e6f1161","Type":"ContainerStarted","Data":"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b"} Nov 03 20:11:46.808713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.808675 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" event={"ID":"2ea8e0cd-ff84-484e-adad-c3552e6f1161","Type":"ContainerStarted","Data":"6da9ce9c524c010d5e94749de1bfdde513755a2aab0fa7a0caeba8dce6c3099c"} Nov 03 20:11:46.810828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.810799 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:46.834400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:46.834352 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podStartSLOduration=1.834338018 podStartE2EDuration="1.834338018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:46.833017943 +0000 UTC m=+847.558025540" watchObservedRunningTime="2026-11-03 20:11:46.834338018 +0000 UTC m=+847.559345614" Nov 03 20:11:47.239298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:47.239254 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:47.240997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:47.240971 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:47.241156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:47.241025 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:47.263069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:47.263038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:47.638635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:47.638598 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:48.238565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:48.238520 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:48.238752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:48.238626 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:48.425135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:48.424511 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:49.238587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:49.238551 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:49.239067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:49.238606 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:49.309402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:49.309371 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.135132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:50.134041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.238515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:50.238478 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:50.238701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:50.238541 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:50.939017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:50.938957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:51.238688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:51.238591 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:51.238688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:51.238653 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:51.830767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:51.830707 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:52.238603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:52.238562 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:11:52.238813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:52.238629 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 20:11:52.705458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:52.705418 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:53.337191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:53.337144 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:53.338338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:53.338317 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:53.342638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:53.342579 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:11:53.687738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:53.687706 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:54.504981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:54.504949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.295306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:55.295247 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.993004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:55.992948 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.791441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:56.791404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:57.594472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:57.594443 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:58.295423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:58.295385 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.082556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:59.082495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.805965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:59.805929 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.937376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:11:59.937342 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:12:00.259458 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:12:00.259421 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:12:00.508687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:00.508657 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.036123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.036078 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:12:01.040243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.040223 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.045384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.045357 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a6f98b-openshift-default-dockercfg-xrwzw\"" Nov 03 20:12:01.052941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.052911 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:12:01.165656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165621 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.165834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165681 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62s8\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.165834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165759 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.165834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165816 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.165943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165897 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.165976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165958 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.166009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.165979 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.166096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.166083 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.166159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.166147 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267157 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267200 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267256 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267289 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267336 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267394 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62s8\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267527 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267660 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267690 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.267757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267732 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.268192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267689 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.268192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.267982 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.268192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.268125 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.268378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.268357 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.269964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.269940 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.270265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.270246 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.280675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.280634 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62s8\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.280931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.280829 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.355802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.355769 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:01.391847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.391799 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.642689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.642659 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:12:01.644675 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:01.644646 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f978df8_2422_4339_a4f0_6dd30f39b9c0.slice/crio-31922f686cf4339521c121560840b4a439224b51817d3193f6f9986684b58102 WatchSource:0}: Error finding container 31922f686cf4339521c121560840b4a439224b51817d3193f6f9986684b58102: Status 404 returned error can't find the container with id 31922f686cf4339521c121560840b4a439224b51817d3193f6f9986684b58102 Nov 03 20:12:01.647034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.647000 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:01.647101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.647070 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:01.647101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.647119 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:01.945367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.945270 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" event={"ID":"2f978df8-2422-4339-a4f0-6dd30f39b9c0","Type":"ContainerStarted","Data":"3f75dc601680c479b1d7013923f8a107819a4fdfcc70062013447133b30cf1a6"} Nov 03 20:12:01.945367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.945339 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" event={"ID":"2f978df8-2422-4339-a4f0-6dd30f39b9c0","Type":"ContainerStarted","Data":"31922f686cf4339521c121560840b4a439224b51817d3193f6f9986684b58102"} Nov 03 20:12:01.972266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:01.972213 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podStartSLOduration=0.972197812 podStartE2EDuration="972.197812ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:01.970751223 +0000 UTC m=+862.695758821" watchObservedRunningTime="2026-11-03 20:12:01.972197812 +0000 UTC m=+862.697205408" Nov 03 20:12:02.280840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:02.280747 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:02.370166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:02.366351 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:02.370166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:02.366483 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:02.370166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:02.366527 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:03.176383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.176324 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.356863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.356821 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:03.357331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.356875 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:03.695889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.695853 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b"] Nov 03 20:12:03.700064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.700043 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.706147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.706098 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:03.706807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.706776 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:03.711308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.710820 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b"] Nov 03 20:12:03.800083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.800039 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.800291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.800215 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.800352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.800294 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzs4x\" (UniqueName: \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.901294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.901261 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.901477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.901352 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.901477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.901412 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzs4x\" (UniqueName: \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.902033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.902008 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.903747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.903720 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.935130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.934059 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzs4x\" (UniqueName: \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\") pod \"maas-api-revoke-sub-f8df55c8-g9g6b\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:03.978611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:03.978523 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:04.053761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.052400 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:04.357237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.357191 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:04.357625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.357256 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:04.379763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.379733 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b"] Nov 03 20:12:04.381805 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:04.381775 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7607488b_e33c_4113_a22d_d4c8426a827d.slice/crio-ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804 WatchSource:0}: Error finding container ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804: Status 404 returned error can't find the container with id ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804 Nov 03 20:12:04.798362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.798269 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:04.964669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:04.964628 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" event={"ID":"7607488b-e33c-4113-a22d-d4c8426a827d","Type":"ContainerStarted","Data":"ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804"} Nov 03 20:12:05.356980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:05.356939 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:05.357231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:05.357005 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:05.491518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:05.491486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.289373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:06.289339 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.360135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:06.357606 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:06.360135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:06.357683 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:07.183852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.183821 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:07.356389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.356350 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:12:07.356559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.356409 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 20:12:07.904201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.904161 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf"] Nov 03 20:12:07.909476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.909450 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:07.915559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:07.915534 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf"] Nov 03 20:12:08.077589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.077554 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.077743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.077659 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.077743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.077717 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tlp\" (UniqueName: \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.165607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.165519 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:08.178943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.178911 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.179183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.178977 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.179183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.179028 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tlp\" (UniqueName: \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.179693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.179674 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.181387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.181367 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.194614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.194589 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tlp\" (UniqueName: \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\") pod \"maas-api-revoke-sub-ac3c58ab-vnkpf\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.230385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.230343 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:08.481255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.481165 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:08.482396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.482333 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:08.483509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.483484 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:12:08.590093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.590057 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf"] Nov 03 20:12:08.590288 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:08.590265 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0382582e_778d_407b_a032_4e5cab1841f4.slice/crio-44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329 WatchSource:0}: Error finding container 44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329: Status 404 returned error can't find the container with id 44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329 Nov 03 20:12:08.991872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:08.991831 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" event={"ID":"0382582e-778d-407b-a032-4e5cab1841f4","Type":"ContainerStarted","Data":"44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329"} Nov 03 20:12:09.208943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:09.208901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:10.000424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.000330 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" event={"ID":"7607488b-e33c-4113-a22d-d4c8426a827d","Type":"ContainerStarted","Data":"97817a091b735ac3d184f402fd0e11b4f316345e5d281792163a4f239a42e93b"} Nov 03 20:12:10.021434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.021372 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" podStartSLOduration=1.999573635 podStartE2EDuration="7.021354946s" totalImagesPullingTime="5.021781311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:10.018499463 +0000 UTC m=+870.743507064" watchObservedRunningTime="2026-11-03 20:12:10.021354946 +0000 UTC m=+870.746362545" Nov 03 20:12:10.045159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.045072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:10.363505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.363473 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj"] Nov 03 20:12:10.367603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.367586 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.376193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.376168 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj"] Nov 03 20:12:10.531183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.531152 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.531367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.531322 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.531482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.531462 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp988\" (UniqueName: \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.632405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.632372 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.632528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.632493 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp988\" (UniqueName: \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.632602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.632583 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.633393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.633370 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.634775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.634754 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.644678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.644654 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp988\" (UniqueName: \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\") pod \"maas-api-revoke-sub-d0c465c9-tp8bj\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.683209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.683184 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:10.887436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:10.887345 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:11.001397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.001367 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj"] Nov 03 20:12:11.003346 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:11.003317 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab2a3ea_22fd_49a3_9e03_03b95075a272.slice/crio-c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b WatchSource:0}: Error finding container c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b: Status 404 returned error can't find the container with id c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b Nov 03 20:12:11.008766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.008671 5954 generic.go:356] "Generic (PLEG): container finished" podID="0382582e-778d-407b-a032-4e5cab1841f4" containerID="c08d7d6c544f979e4eb090aca252f6ecde59932bb3917c4e7a0607406884b2c1" exitCode=0 Nov 03 20:12:11.008766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.008714 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" event={"ID":"0382582e-778d-407b-a032-4e5cab1841f4","Type":"ContainerDied","Data":"c08d7d6c544f979e4eb090aca252f6ecde59932bb3917c4e7a0607406884b2c1"} Nov 03 20:12:11.010488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.010466 5954 generic.go:356] "Generic (PLEG): container finished" podID="7607488b-e33c-4113-a22d-d4c8426a827d" containerID="97817a091b735ac3d184f402fd0e11b4f316345e5d281792163a4f239a42e93b" exitCode=0 Nov 03 20:12:11.010570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.010529 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" event={"ID":"7607488b-e33c-4113-a22d-d4c8426a827d","Type":"ContainerDied","Data":"97817a091b735ac3d184f402fd0e11b4f316345e5d281792163a4f239a42e93b"} Nov 03 20:12:11.678628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:11.678598 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:12.018721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.018627 5954 generic.go:356] "Generic (PLEG): container finished" podID="bab2a3ea-22fd-49a3-9e03-03b95075a272" containerID="e3d24ea454e271b0803645b3927ed6a2d03ed32e92341a1e11c215f3499e9af5" exitCode=0 Nov 03 20:12:12.018877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.018708 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" event={"ID":"bab2a3ea-22fd-49a3-9e03-03b95075a272","Type":"ContainerDied","Data":"e3d24ea454e271b0803645b3927ed6a2d03ed32e92341a1e11c215f3499e9af5"} Nov 03 20:12:12.018877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.018767 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" event={"ID":"bab2a3ea-22fd-49a3-9e03-03b95075a272","Type":"ContainerStarted","Data":"c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b"} Nov 03 20:12:12.194816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.194789 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:12.232881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.232839 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:12.368333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368297 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\" (UniqueName: \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\") pod \"0382582e-778d-407b-a032-4e5cab1841f4\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " Nov 03 20:12:12.368508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368346 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\" (UniqueName: \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\") pod \"7607488b-e33c-4113-a22d-d4c8426a827d\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " Nov 03 20:12:12.368508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368465 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\") pod \"0382582e-778d-407b-a032-4e5cab1841f4\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " Nov 03 20:12:12.368508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368490 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\") pod \"7607488b-e33c-4113-a22d-d4c8426a827d\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " Nov 03 20:12:12.368665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368509 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\") pod \"7607488b-e33c-4113-a22d-d4c8426a827d\" (UID: \"7607488b-e33c-4113-a22d-d4c8426a827d\") " Nov 03 20:12:12.368665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368528 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\") pod \"0382582e-778d-407b-a032-4e5cab1841f4\" (UID: \"0382582e-778d-407b-a032-4e5cab1841f4\") " Nov 03 20:12:12.368923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368892 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca" pod "7607488b-e33c-4113-a22d-d4c8426a827d" (UID: "7607488b-e33c-4113-a22d-d4c8426a827d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:12.369052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.368936 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca" pod "0382582e-778d-407b-a032-4e5cab1841f4" (UID: "0382582e-778d-407b-a032-4e5cab1841f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:12.369171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.369152 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.369230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.369177 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.370777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.370749 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca" pod "7607488b-e33c-4113-a22d-d4c8426a827d" (UID: "7607488b-e33c-4113-a22d-d4c8426a827d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:12.370777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.370749 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp" pod "0382582e-778d-407b-a032-4e5cab1841f4" (UID: "0382582e-778d-407b-a032-4e5cab1841f4"). InnerVolumeSpecName "kube-api-access-v2tlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:12.370892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.370777 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca" pod "0382582e-778d-407b-a032-4e5cab1841f4" (UID: "0382582e-778d-407b-a032-4e5cab1841f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:12.370892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.370800 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x" pod "7607488b-e33c-4113-a22d-d4c8426a827d" (UID: "7607488b-e33c-4113-a22d-d4c8426a827d"). InnerVolumeSpecName "kube-api-access-dzs4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:12.394295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.394261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:12.475205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.470988 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tlp\" (UniqueName: \"kubernetes.io/projected/0382582e-778d-407b-a032-4e5cab1841f4-kube-api-access-v2tlp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.475205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.471034 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzs4x\" (UniqueName: \"kubernetes.io/projected/7607488b-e33c-4113-a22d-d4c8426a827d-kube-api-access-dzs4x\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.475205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.471049 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7607488b-e33c-4113-a22d-d4c8426a827d-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.475205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:12.471065 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0382582e-778d-407b-a032-4e5cab1841f4-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.027433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.027349 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" Nov 03 20:12:13.027433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.027358 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf" event={"ID":"0382582e-778d-407b-a032-4e5cab1841f4","Type":"ContainerDied","Data":"44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329"} Nov 03 20:12:13.027433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.027415 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44fa3a2cded0735c2041d0d010b601fc9f81dc8ba1744ea06328adc7284bc329" Nov 03 20:12:13.028920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.028890 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" event={"ID":"7607488b-e33c-4113-a22d-d4c8426a827d","Type":"ContainerDied","Data":"ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804"} Nov 03 20:12:13.028920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.028918 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b" Nov 03 20:12:13.029081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.028944 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef996f3ba459e11696ff01e11e21d2510c639c51edf1dee4a3add78d3bf53804" Nov 03 20:12:13.098531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.098498 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:13.246374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.246341 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:13.386606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.386571 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\") pod \"bab2a3ea-22fd-49a3-9e03-03b95075a272\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " Nov 03 20:12:13.386606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.386620 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\") pod \"bab2a3ea-22fd-49a3-9e03-03b95075a272\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " Nov 03 20:12:13.386894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.386639 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\" (UniqueName: \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\") pod \"bab2a3ea-22fd-49a3-9e03-03b95075a272\" (UID: \"bab2a3ea-22fd-49a3-9e03-03b95075a272\") " Nov 03 20:12:13.387096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.387070 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca" pod "bab2a3ea-22fd-49a3-9e03-03b95075a272" (UID: "bab2a3ea-22fd-49a3-9e03-03b95075a272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:13.387204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.387185 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.388869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.388839 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca" pod "bab2a3ea-22fd-49a3-9e03-03b95075a272" (UID: "bab2a3ea-22fd-49a3-9e03-03b95075a272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:13.388981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.388883 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988" pod "bab2a3ea-22fd-49a3-9e03-03b95075a272" (UID: "bab2a3ea-22fd-49a3-9e03-03b95075a272"). InnerVolumeSpecName "kube-api-access-bp988". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:13.488752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.488718 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab2a3ea-22fd-49a3-9e03-03b95075a272-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.488752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:13.488750 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp988\" (UniqueName: \"kubernetes.io/projected/bab2a3ea-22fd-49a3-9e03-03b95075a272-kube-api-access-bp988\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:14.057992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:14.057952 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:14.069469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:14.069439 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" Nov 03 20:12:14.069469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:14.069452 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj" event={"ID":"bab2a3ea-22fd-49a3-9e03-03b95075a272","Type":"ContainerDied","Data":"c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b"} Nov 03 20:12:14.069682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:14.069516 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03e25d779a8b07e3ab295c609b5ceac41f4bb3ff565ad3ebdf20f6717d4232b" Nov 03 20:12:14.791684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:14.791637 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:15.497543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:15.497514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:16.305445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:16.305409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:17.075877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.075846 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:17.886874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.886845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:17.915906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.915873 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5"] Nov 03 20:12:17.916529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916509 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7607488b-e33c-4113-a22d-d4c8426a827d" containerName="revoke-keys" Nov 03 20:12:17.916529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916529 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7607488b-e33c-4113-a22d-d4c8426a827d" containerName="revoke-keys" Nov 03 20:12:17.916699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916606 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0382582e-778d-407b-a032-4e5cab1841f4" containerName="revoke-keys" Nov 03 20:12:17.916699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916616 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0382582e-778d-407b-a032-4e5cab1841f4" containerName="revoke-keys" Nov 03 20:12:17.916699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916628 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bab2a3ea-22fd-49a3-9e03-03b95075a272" containerName="revoke-keys" Nov 03 20:12:17.916699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916635 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab2a3ea-22fd-49a3-9e03-03b95075a272" containerName="revoke-keys" Nov 03 20:12:17.916890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916813 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0382582e-778d-407b-a032-4e5cab1841f4" containerName="revoke-keys" Nov 03 20:12:17.916890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916828 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab2a3ea-22fd-49a3-9e03-03b95075a272" containerName="revoke-keys" Nov 03 20:12:17.916890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.916844 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7607488b-e33c-4113-a22d-d4c8426a827d" containerName="revoke-keys" Nov 03 20:12:17.921154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.921134 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:17.927651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.927543 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:17.945127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.944904 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:17.965131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.954074 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5"] Nov 03 20:12:17.977314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.971262 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:17.983664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.983630 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:17.983822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:17.983744 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99kf\" (UniqueName: \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.084529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.084482 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.084974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.084552 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.084974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.084626 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99kf\" (UniqueName: \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.085274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.085252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.086978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.086953 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.097863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.097840 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99kf\" (UniqueName: \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\") pod \"maas-api-revoke-sub-221c3325-62mp5\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.276160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.276059 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:18.554071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.554047 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5"] Nov 03 20:12:18.556006 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:18.555980 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cae5a1b_6fe5_4e3f_ae87_7d896df88eb3.slice/crio-b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc WatchSource:0}: Error finding container b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc: Status 404 returned error can't find the container with id b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc Nov 03 20:12:18.685359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:18.685327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:19.102275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:19.102239 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" event={"ID":"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3","Type":"ContainerStarted","Data":"661239ced010ad38eb3b34262cf37507406f84a5c4cdc4bfc1aac459c531924e"} Nov 03 20:12:19.102682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:19.102293 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" event={"ID":"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3","Type":"ContainerStarted","Data":"b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc"} Nov 03 20:12:19.127320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:19.127267 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" podStartSLOduration=2.127250396 podStartE2EDuration="2.127250396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:19.123234107 +0000 UTC m=+879.848241703" watchObservedRunningTime="2026-11-03 20:12:19.127250396 +0000 UTC m=+879.852257991" Nov 03 20:12:19.752772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:19.752682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:20.110374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:20.110340 5954 generic.go:356] "Generic (PLEG): container finished" podID="8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" containerID="661239ced010ad38eb3b34262cf37507406f84a5c4cdc4bfc1aac459c531924e" exitCode=0 Nov 03 20:12:20.110753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:20.110404 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" event={"ID":"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3","Type":"ContainerDied","Data":"661239ced010ad38eb3b34262cf37507406f84a5c4cdc4bfc1aac459c531924e"} Nov 03 20:12:20.701556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:20.701521 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:21.301505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.301478 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:21.345571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.345533 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\") pod \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " Nov 03 20:12:21.345758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.345599 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\") pod \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " Nov 03 20:12:21.345758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.345666 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\" (UniqueName: \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\") pod \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\" (UID: \"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3\") " Nov 03 20:12:21.345951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.345929 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca" pod "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" (UID: "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:21.346242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.346218 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:21.348386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.348360 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca" pod "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" (UID: "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:21.349170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.349144 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf" pod "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" (UID: "8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3"). InnerVolumeSpecName "kube-api-access-q99kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:21.447622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.447593 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:21.447622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.447618 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99kf\" (UniqueName: \"kubernetes.io/projected/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3-kube-api-access-q99kf\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:21.585719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:21.585687 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:22.126170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.126139 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" Nov 03 20:12:22.126340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.126165 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5" event={"ID":"8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3","Type":"ContainerDied","Data":"b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc"} Nov 03 20:12:22.126340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.126221 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7698b785a52690d93ca476f4199c9ae62196beae74d4ab8ffc032f2ff83f9bc" Nov 03 20:12:22.294942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.294898 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f"] Nov 03 20:12:22.295573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.295554 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" containerName="revoke-keys" Nov 03 20:12:22.295670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.295575 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" containerName="revoke-keys" Nov 03 20:12:22.295760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.295748 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" containerName="revoke-keys" Nov 03 20:12:22.299329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.299309 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.304319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.304281 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:22.304686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.304405 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:22.307711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.307685 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f"] Nov 03 20:12:22.359555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.359519 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.359742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.359563 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4j9p\" (UniqueName: \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.359742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.359581 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.444333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.444249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:22.461614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.461581 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.461795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.461647 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4j9p\" (UniqueName: \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.461795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.461680 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.462365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.462339 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.463921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.463902 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.473617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.473591 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4j9p\" (UniqueName: \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\") pod \"maas-api-revoke-sub-09551f22-g5d7f\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.615839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.615808 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:22.902141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:22.902098 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f"] Nov 03 20:12:22.903190 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:22.903161 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01260483_d289_488b_a38b_6b2c9166f290.slice/crio-6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8 WatchSource:0}: Error finding container 6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8: Status 404 returned error can't find the container with id 6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8 Nov 03 20:12:23.134141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.134093 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" event={"ID":"01260483-d289-488b-a38b-6b2c9166f290","Type":"ContainerStarted","Data":"6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8"} Nov 03 20:12:23.297797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.297754 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:23.790508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.790475 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n"] Nov 03 20:12:23.794408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.794389 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.803201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.803177 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n"] Nov 03 20:12:23.881025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.880994 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.881193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.881045 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.881314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.881286 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mqz\" (UniqueName: \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.983121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.983020 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mqz\" (UniqueName: \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.983270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.983122 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.983270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.983154 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.983797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.983778 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.985551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.985526 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:23.998241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:23.998214 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mqz\" (UniqueName: \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\") pod \"maas-api-revoke-sub-63d060ba-v845n\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:24.119538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:24.119503 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:24.142816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:24.142782 5954 generic.go:356] "Generic (PLEG): container finished" podID="01260483-d289-488b-a38b-6b2c9166f290" containerID="1ce1c57e515b4d35f9edd902ea065726860f87f1c34d214fd3b1ceaa19f7be62" exitCode=0 Nov 03 20:12:24.142964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:24.142832 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" event={"ID":"01260483-d289-488b-a38b-6b2c9166f290","Type":"ContainerDied","Data":"1ce1c57e515b4d35f9edd902ea065726860f87f1c34d214fd3b1ceaa19f7be62"} Nov 03 20:12:24.273681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:24.273596 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.398935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:24.398910 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n"] Nov 03 20:12:24.401125 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:24.401092 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e47058_56b5_4088_872d_3c4166ee19b8.slice/crio-0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85 WatchSource:0}: Error finding container 0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85: Status 404 returned error can't find the container with id 0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85 Nov 03 20:12:25.075149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.075023 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:25.163385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.163242 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" event={"ID":"71e47058-56b5-4088-872d-3c4166ee19b8","Type":"ContainerStarted","Data":"198f262aabdb92689772531892557263e0d1e7e15de1530acd3e98633de2280c"} Nov 03 20:12:25.163385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.163339 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" event={"ID":"71e47058-56b5-4088-872d-3c4166ee19b8","Type":"ContainerStarted","Data":"0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85"} Nov 03 20:12:25.186046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.185997 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" podStartSLOduration=2.185983093 podStartE2EDuration="2.185983093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:25.183831573 +0000 UTC m=+885.908839169" watchObservedRunningTime="2026-11-03 20:12:25.185983093 +0000 UTC m=+885.910990688" Nov 03 20:12:25.312617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.312593 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:25.401543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.401514 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\") pod \"01260483-d289-488b-a38b-6b2c9166f290\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " Nov 03 20:12:25.401704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.401551 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\") pod \"01260483-d289-488b-a38b-6b2c9166f290\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " Nov 03 20:12:25.401704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.401663 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\" (UniqueName: \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\") pod \"01260483-d289-488b-a38b-6b2c9166f290\" (UID: \"01260483-d289-488b-a38b-6b2c9166f290\") " Nov 03 20:12:25.401961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.401931 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca" pod "01260483-d289-488b-a38b-6b2c9166f290" (UID: "01260483-d289-488b-a38b-6b2c9166f290"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:25.403725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.403700 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p" pod "01260483-d289-488b-a38b-6b2c9166f290" (UID: "01260483-d289-488b-a38b-6b2c9166f290"). InnerVolumeSpecName "kube-api-access-x4j9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:25.403960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.403935 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca" pod "01260483-d289-488b-a38b-6b2c9166f290" (UID: "01260483-d289-488b-a38b-6b2c9166f290"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:25.503298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.503263 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4j9p\" (UniqueName: \"kubernetes.io/projected/01260483-d289-488b-a38b-6b2c9166f290-kube-api-access-x4j9p\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.503298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.503292 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01260483-d289-488b-a38b-6b2c9166f290-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.503298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.503305 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01260483-d289-488b-a38b-6b2c9166f290-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.885547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:25.885513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:26.169733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.169647 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" Nov 03 20:12:26.169733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.169679 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f" event={"ID":"01260483-d289-488b-a38b-6b2c9166f290","Type":"ContainerDied","Data":"6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8"} Nov 03 20:12:26.170202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.169733 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fa0659e3801e664fd55ba3257755d27acb1f55df0b44cbbd22dd4c28b9486a8" Nov 03 20:12:26.171212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.171185 5954 generic.go:356] "Generic (PLEG): container finished" podID="71e47058-56b5-4088-872d-3c4166ee19b8" containerID="198f262aabdb92689772531892557263e0d1e7e15de1530acd3e98633de2280c" exitCode=0 Nov 03 20:12:26.171340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.171262 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" event={"ID":"71e47058-56b5-4088-872d-3c4166ee19b8","Type":"ContainerDied","Data":"198f262aabdb92689772531892557263e0d1e7e15de1530acd3e98633de2280c"} Nov 03 20:12:26.584770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:26.584690 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.299533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.299500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.328246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.328220 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:27.424825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.424793 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\" (UniqueName: \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\") pod \"71e47058-56b5-4088-872d-3c4166ee19b8\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " Nov 03 20:12:27.425010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.424977 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\") pod \"71e47058-56b5-4088-872d-3c4166ee19b8\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " Nov 03 20:12:27.425081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.425047 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\") pod \"71e47058-56b5-4088-872d-3c4166ee19b8\" (UID: \"71e47058-56b5-4088-872d-3c4166ee19b8\") " Nov 03 20:12:27.425548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.425512 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca" pod "71e47058-56b5-4088-872d-3c4166ee19b8" (UID: "71e47058-56b5-4088-872d-3c4166ee19b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:27.427070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.427036 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca" pod "71e47058-56b5-4088-872d-3c4166ee19b8" (UID: "71e47058-56b5-4088-872d-3c4166ee19b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:27.427474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.427457 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz" pod "71e47058-56b5-4088-872d-3c4166ee19b8" (UID: "71e47058-56b5-4088-872d-3c4166ee19b8"). InnerVolumeSpecName "kube-api-access-q8mqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:27.527174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.527099 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.527174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.527173 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e47058-56b5-4088-872d-3c4166ee19b8-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.527369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:27.527186 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mqz\" (UniqueName: \"kubernetes.io/projected/71e47058-56b5-4088-872d-3c4166ee19b8-kube-api-access-q8mqz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:28.175187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:28.175143 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.186179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:28.186153 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" Nov 03 20:12:28.186351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:28.186142 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n" event={"ID":"71e47058-56b5-4088-872d-3c4166ee19b8","Type":"ContainerDied","Data":"0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85"} Nov 03 20:12:28.186351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:28.186256 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf8559e90b0bb3daba1d7f8b483ed26d49147fa17f8ad54112d5c74f8095e85" Nov 03 20:12:28.379156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:28.379120 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:12:29.280747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:29.280690 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:30.452743 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:12:30.452698 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:12:31.231133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:31.227092 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.493808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.493772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.818200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.817264 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.961129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.960151 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk"] Nov 03 20:12:32.961129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961092 5954 cpu_manager.go:414] "containerMap: removing container" podUID="01260483-d289-488b-a38b-6b2c9166f290" containerName="revoke-keys" Nov 03 20:12:32.961129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961128 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01260483-d289-488b-a38b-6b2c9166f290" containerName="revoke-keys" Nov 03 20:12:32.961129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961143 5954 cpu_manager.go:414] "containerMap: removing container" podUID="71e47058-56b5-4088-872d-3c4166ee19b8" containerName="revoke-keys" Nov 03 20:12:32.961482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961150 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e47058-56b5-4088-872d-3c4166ee19b8" containerName="revoke-keys" Nov 03 20:12:32.961482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961339 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="01260483-d289-488b-a38b-6b2c9166f290" containerName="revoke-keys" Nov 03 20:12:32.961482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:32.961353 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e47058-56b5-4088-872d-3c4166ee19b8" containerName="revoke-keys" Nov 03 20:12:33.099775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.097787 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk"] Nov 03 20:12:33.099775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.097917 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.103860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.103787 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:33.104433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.104397 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:33.191957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.191867 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.196419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.196379 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.196764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.196743 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x94n\" (UniqueName: \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.301168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.297824 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.301168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.297916 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x94n\" (UniqueName: \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.301168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.298059 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.301168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.298955 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.304283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.304249 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.313325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.313259 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x94n\" (UniqueName: \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\") pod \"maas-api-revoke-sub-8c42a4a4-wrmvk\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.463128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.462310 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:33.595068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.595018 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.968043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:33.968014 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk"] Nov 03 20:12:33.972017 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:33.971047 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53ac0e3_b9b8_4f68_a885_6fe93084a936.slice/crio-b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8 WatchSource:0}: Error finding container b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8: Status 404 returned error can't find the container with id b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8 Nov 03 20:12:34.116782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.116743 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf"] Nov 03 20:12:34.121990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.121968 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.129726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.129699 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf"] Nov 03 20:12:34.191238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.191205 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfsp\" (UniqueName: \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.191238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.191242 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.191458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.191370 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.293287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.293190 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfsp\" (UniqueName: \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.293287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.293238 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.293530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.293312 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.293915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.293893 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.295680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.295658 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.304763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.304740 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfsp\" (UniqueName: \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\") pod \"maas-api-revoke-sub-b07af413-nxlpf\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.448919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.448886 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:34.450507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.450477 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" event={"ID":"c53ac0e3-b9b8-4f68-a885-6fe93084a936","Type":"ContainerStarted","Data":"8481b61dda3b2efb3b790695c4cd2861afa7b46700068a56b5f0d8912c79d3ac"} Nov 03 20:12:34.450639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.450532 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" event={"ID":"c53ac0e3-b9b8-4f68-a885-6fe93084a936","Type":"ContainerStarted","Data":"b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8"} Nov 03 20:12:34.470022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.469960 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" podStartSLOduration=2.469938541 podStartE2EDuration="2.469938541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.468890903 +0000 UTC m=+895.193898501" watchObservedRunningTime="2026-11-03 20:12:34.469938541 +0000 UTC m=+895.194946139" Nov 03 20:12:34.543544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.543458 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:34.753579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:34.753544 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf"] Nov 03 20:12:34.755181 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:34.755154 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2feed9_d5a0_404a_9291_230f0985aeb9.slice/crio-ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3 WatchSource:0}: Error finding container ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3: Status 404 returned error can't find the container with id ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3 Nov 03 20:12:35.458679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.458644 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" event={"ID":"9a2feed9-d5a0-404a-9291-230f0985aeb9","Type":"ContainerStarted","Data":"13336541bda4e8f40cf88a711b1b19abad65eae02262bb0196d0ec2c55ff7338"} Nov 03 20:12:35.458859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.458708 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" event={"ID":"9a2feed9-d5a0-404a-9291-230f0985aeb9","Type":"ContainerStarted","Data":"ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3"} Nov 03 20:12:35.460173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.460143 5954 generic.go:356] "Generic (PLEG): container finished" podID="c53ac0e3-b9b8-4f68-a885-6fe93084a936" containerID="8481b61dda3b2efb3b790695c4cd2861afa7b46700068a56b5f0d8912c79d3ac" exitCode=0 Nov 03 20:12:35.460281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.460219 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" event={"ID":"c53ac0e3-b9b8-4f68-a885-6fe93084a936","Type":"ContainerDied","Data":"8481b61dda3b2efb3b790695c4cd2861afa7b46700068a56b5f0d8912c79d3ac"} Nov 03 20:12:35.478130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.478059 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" podStartSLOduration=1.478031662 podStartE2EDuration="1.478031662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:35.476850889 +0000 UTC m=+896.201858491" watchObservedRunningTime="2026-11-03 20:12:35.478031662 +0000 UTC m=+896.203039258" Nov 03 20:12:35.624967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:35.624943 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:36.468246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.468207 5954 generic.go:356] "Generic (PLEG): container finished" podID="9a2feed9-d5a0-404a-9291-230f0985aeb9" containerID="13336541bda4e8f40cf88a711b1b19abad65eae02262bb0196d0ec2c55ff7338" exitCode=0 Nov 03 20:12:36.468667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.468297 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" event={"ID":"9a2feed9-d5a0-404a-9291-230f0985aeb9","Type":"ContainerDied","Data":"13336541bda4e8f40cf88a711b1b19abad65eae02262bb0196d0ec2c55ff7338"} Nov 03 20:12:36.495385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.495326 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:36.666903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.666874 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:36.795358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.795273 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\") pod \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " Nov 03 20:12:36.795358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.795317 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\") pod \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " Nov 03 20:12:36.795551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.795462 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\" (UniqueName: \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\") pod \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\" (UID: \"c53ac0e3-b9b8-4f68-a885-6fe93084a936\") " Nov 03 20:12:36.795652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.795629 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca" pod "c53ac0e3-b9b8-4f68-a885-6fe93084a936" (UID: "c53ac0e3-b9b8-4f68-a885-6fe93084a936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:36.795867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.795840 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:36.797424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.797402 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca" pod "c53ac0e3-b9b8-4f68-a885-6fe93084a936" (UID: "c53ac0e3-b9b8-4f68-a885-6fe93084a936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:36.797515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.797493 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n" pod "c53ac0e3-b9b8-4f68-a885-6fe93084a936" (UID: "c53ac0e3-b9b8-4f68-a885-6fe93084a936"). InnerVolumeSpecName "kube-api-access-5x94n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:36.897513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.897475 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53ac0e3-b9b8-4f68-a885-6fe93084a936-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:36.897513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:36.897507 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x94n\" (UniqueName: \"kubernetes.io/projected/c53ac0e3-b9b8-4f68-a885-6fe93084a936-kube-api-access-5x94n\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.328447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.328413 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc"] Nov 03 20:12:37.328991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.328978 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c53ac0e3-b9b8-4f68-a885-6fe93084a936" containerName="revoke-keys" Nov 03 20:12:37.329051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.328993 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53ac0e3-b9b8-4f68-a885-6fe93084a936" containerName="revoke-keys" Nov 03 20:12:37.329121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.329098 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53ac0e3-b9b8-4f68-a885-6fe93084a936" containerName="revoke-keys" Nov 03 20:12:37.332500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.332484 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.341848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.341824 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc"] Nov 03 20:12:37.402946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.402912 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.403150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.403134 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltcl\" (UniqueName: \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.403230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.403173 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.476072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.476043 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" Nov 03 20:12:37.476505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.476075 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk" event={"ID":"c53ac0e3-b9b8-4f68-a885-6fe93084a936","Type":"ContainerDied","Data":"b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8"} Nov 03 20:12:37.476505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.476134 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b1129eeb3f1c62e547f8b976412b1ffdc5ec619d540c04aecd8fb6e6cb9ec8" Nov 03 20:12:37.489069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.489031 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:37.504914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.504870 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.505136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.505036 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.505381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.505358 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltcl\" (UniqueName: \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.505689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.505660 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.508168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.508142 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.523711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.523683 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltcl\" (UniqueName: \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\") pod \"maas-api-revoke-sub-b1aa1299-lghvc\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.647664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.647638 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:37.692063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.692034 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:37.707746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.707720 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\" (UniqueName: \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\") pod \"9a2feed9-d5a0-404a-9291-230f0985aeb9\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " Nov 03 20:12:37.707847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.707796 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\") pod \"9a2feed9-d5a0-404a-9291-230f0985aeb9\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " Nov 03 20:12:37.707888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.707869 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\") pod \"9a2feed9-d5a0-404a-9291-230f0985aeb9\" (UID: \"9a2feed9-d5a0-404a-9291-230f0985aeb9\") " Nov 03 20:12:37.708555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.708514 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca" pod "9a2feed9-d5a0-404a-9291-230f0985aeb9" (UID: "9a2feed9-d5a0-404a-9291-230f0985aeb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:37.716239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.710710 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp" pod "9a2feed9-d5a0-404a-9291-230f0985aeb9" (UID: "9a2feed9-d5a0-404a-9291-230f0985aeb9"). InnerVolumeSpecName "kube-api-access-7dfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:37.718524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.718462 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca" pod "9a2feed9-d5a0-404a-9291-230f0985aeb9" (UID: "9a2feed9-d5a0-404a-9291-230f0985aeb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:37.809209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.809173 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.809209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.809202 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a2feed9-d5a0-404a-9291-230f0985aeb9-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.809209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.809211 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfsp\" (UniqueName: \"kubernetes.io/projected/9a2feed9-d5a0-404a-9291-230f0985aeb9-kube-api-access-7dfsp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.934920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:37.934888 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc"] Nov 03 20:12:37.936257 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:37.936231 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275a3b30_a5c1_46b9_8cbd_9c7149e87878.slice/crio-c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5 WatchSource:0}: Error finding container c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5: Status 404 returned error can't find the container with id c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5 Nov 03 20:12:38.319780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.319745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:38.484554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.484445 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" event={"ID":"275a3b30-a5c1-46b9-8cbd-9c7149e87878","Type":"ContainerStarted","Data":"1672e6f974a5c751d15dd565bab82e2fa0c473ce50bd8275dc5e4252bebc8e11"} Nov 03 20:12:38.484554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.484514 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" event={"ID":"275a3b30-a5c1-46b9-8cbd-9c7149e87878","Type":"ContainerStarted","Data":"c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5"} Nov 03 20:12:38.485747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.485726 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" event={"ID":"9a2feed9-d5a0-404a-9291-230f0985aeb9","Type":"ContainerDied","Data":"ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3"} Nov 03 20:12:38.485747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.485739 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf" Nov 03 20:12:38.485879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.485762 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceacbffbd3e0832c6427238062def93e86a1d09cafca0495cb9abc0a438880c3" Nov 03 20:12:38.507435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:38.507380 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" podStartSLOduration=1.5073655590000001 podStartE2EDuration="1.507365559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:38.504912828 +0000 UTC m=+899.229920424" watchObservedRunningTime="2026-11-03 20:12:38.507365559 +0000 UTC m=+899.232373155" Nov 03 20:12:39.277362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:39.277327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:39.504884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:39.504838 5954 generic.go:356] "Generic (PLEG): container finished" podID="275a3b30-a5c1-46b9-8cbd-9c7149e87878" containerID="1672e6f974a5c751d15dd565bab82e2fa0c473ce50bd8275dc5e4252bebc8e11" exitCode=0 Nov 03 20:12:39.505356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:39.504887 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" event={"ID":"275a3b30-a5c1-46b9-8cbd-9c7149e87878","Type":"ContainerDied","Data":"1672e6f974a5c751d15dd565bab82e2fa0c473ce50bd8275dc5e4252bebc8e11"} Nov 03 20:12:40.090100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.090063 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:40.668669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.668640 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:40.747609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.747569 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\") pod \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " Nov 03 20:12:40.747824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.747639 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\") pod \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " Nov 03 20:12:40.747824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.747703 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\" (UniqueName: \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\") pod \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\" (UID: \"275a3b30-a5c1-46b9-8cbd-9c7149e87878\") " Nov 03 20:12:40.748017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.747987 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca" pod "275a3b30-a5c1-46b9-8cbd-9c7149e87878" (UID: "275a3b30-a5c1-46b9-8cbd-9c7149e87878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:40.748243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.748223 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:40.749906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.749882 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca" pod "275a3b30-a5c1-46b9-8cbd-9c7149e87878" (UID: "275a3b30-a5c1-46b9-8cbd-9c7149e87878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:40.750007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.749916 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl" pod "275a3b30-a5c1-46b9-8cbd-9c7149e87878" (UID: "275a3b30-a5c1-46b9-8cbd-9c7149e87878"). InnerVolumeSpecName "kube-api-access-7ltcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:40.849186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.849086 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275a3b30-a5c1-46b9-8cbd-9c7149e87878-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:40.849186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.849139 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltcl\" (UniqueName: \"kubernetes.io/projected/275a3b30-a5c1-46b9-8cbd-9c7149e87878-kube-api-access-7ltcl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:40.886046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:40.886016 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:41.519389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:41.519355 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" Nov 03 20:12:41.519560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:41.519354 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc" event={"ID":"275a3b30-a5c1-46b9-8cbd-9c7149e87878","Type":"ContainerDied","Data":"c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5"} Nov 03 20:12:41.519560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:41.519480 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7890910fee2e91c13afbb877e97598ffdda590c68d19ef018ef2dc015470bc5" Nov 03 20:12:41.576511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:41.576454 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:42.285540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:42.285507 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:42.687469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:42.687432 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.252515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:43.252459 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:44.592133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:44.584725 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:45.695011 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.694850 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:45.922436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.922392 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:12:45.923395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923311 5954 cpu_manager.go:414] "containerMap: removing container" podUID="275a3b30-a5c1-46b9-8cbd-9c7149e87878" containerName="revoke-keys" Nov 03 20:12:45.923395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923339 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275a3b30-a5c1-46b9-8cbd-9c7149e87878" containerName="revoke-keys" Nov 03 20:12:45.923395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923390 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9a2feed9-d5a0-404a-9291-230f0985aeb9" containerName="revoke-keys" Nov 03 20:12:45.923395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923398 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2feed9-d5a0-404a-9291-230f0985aeb9" containerName="revoke-keys" Nov 03 20:12:45.923782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923577 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="275a3b30-a5c1-46b9-8cbd-9c7149e87878" containerName="revoke-keys" Nov 03 20:12:45.923782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.923597 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2feed9-d5a0-404a-9291-230f0985aeb9" containerName="revoke-keys" Nov 03 20:12:45.938240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.938215 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:12:45.938467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.938340 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998302 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998351 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998482 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998529 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998670 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnn5\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998711 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998754 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998808 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:45.998888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:45.998876 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.099897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.099857 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100116 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.099919 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100116 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.099993 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100116 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100050 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100169 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnn5\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100198 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100220 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100260 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100291 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.100540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100401 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.101002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.100955 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.101142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.101010 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.101432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.101405 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.101780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.101757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.102793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.102768 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.103321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.103300 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.113561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.113534 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnn5\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.113740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.113558 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.256462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.256370 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:46.554051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.554021 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:12:46.556873 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:46.556837 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21debdd0_6616_40ec_8610_e5a6c3149fcc.slice/crio-ddb532404ad3c886460ed5d29bb02c20530ab9e9dde14227809d1ce08fe8f887 WatchSource:0}: Error finding container ddb532404ad3c886460ed5d29bb02c20530ab9e9dde14227809d1ce08fe8f887: Status 404 returned error can't find the container with id ddb532404ad3c886460ed5d29bb02c20530ab9e9dde14227809d1ce08fe8f887 Nov 03 20:12:46.560332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.560290 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.560442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.560384 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.560442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.560429 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.599516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.599492 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:46.636511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.636476 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" event={"ID":"21debdd0-6616-40ec-8610-e5a6c3149fcc","Type":"ContainerStarted","Data":"ddb532404ad3c886460ed5d29bb02c20530ab9e9dde14227809d1ce08fe8f887"} Nov 03 20:12:46.827217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.813872 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p"] Nov 03 20:12:46.828910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.828881 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:46.834245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.834213 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p"] Nov 03 20:12:46.842412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.842387 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:46.842696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.842646 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:46.920971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.920928 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:46.921225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.921208 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr47m\" (UniqueName: \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:46.921348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:46.921324 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.023333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.023274 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.023517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.023399 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.023609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.023592 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr47m\" (UniqueName: \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.024330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.024301 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.026513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.026486 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.036593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.036570 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr47m\" (UniqueName: \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\") pod \"maas-api-revoke-sub-cf606668-69s5p\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.149910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.149861 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:47.166707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.166676 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:47.332560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.332524 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:47.470453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.470421 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p"] Nov 03 20:12:47.473592 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:47.473512 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75711e96_f618_48d7_b7ad_db6f25b4f4f7.slice/crio-32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714 WatchSource:0}: Error finding container 32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714: Status 404 returned error can't find the container with id 32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714 Nov 03 20:12:47.646562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.646526 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" event={"ID":"75711e96-f618-48d7-b7ad-db6f25b4f4f7","Type":"ContainerStarted","Data":"32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714"} Nov 03 20:12:47.648266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.648243 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" event={"ID":"21debdd0-6616-40ec-8610-e5a6c3149fcc","Type":"ContainerStarted","Data":"097478e56839e01a3a0024719943b2d24761c4cd056c3590c427231b2082d19d"} Nov 03 20:12:47.672072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.672014 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podStartSLOduration=2.671995712 podStartE2EDuration="2.671995712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:47.671620929 +0000 UTC m=+908.396628538" watchObservedRunningTime="2026-11-03 20:12:47.671995712 +0000 UTC m=+908.397003310" Nov 03 20:12:47.726193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:47.724562 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:48.256741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:48.256702 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:48.258611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:48.258581 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.258738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:48.258629 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:48.657733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:48.657697 5954 generic.go:356] "Generic (PLEG): container finished" podID="75711e96-f618-48d7-b7ad-db6f25b4f4f7" containerID="be40508cde0ca2fafc238ef1451afe8a1c80034fc2b76601c9f7e9e1e0251e9b" exitCode=0 Nov 03 20:12:48.657933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:48.657808 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" event={"ID":"75711e96-f618-48d7-b7ad-db6f25b4f4f7","Type":"ContainerDied","Data":"be40508cde0ca2fafc238ef1451afe8a1c80034fc2b76601c9f7e9e1e0251e9b"} Nov 03 20:12:49.007519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.007482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:49.256899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.256862 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.257333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.256920 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:49.758241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.757701 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j"] Nov 03 20:12:49.769040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.767473 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.772196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.772155 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j"] Nov 03 20:12:49.842224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.842185 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:49.871240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.871216 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.871421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.871402 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42pgj\" (UniqueName: \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.871530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.871513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.883506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.883484 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:49.974727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.974595 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\" (UniqueName: \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\") pod \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " Nov 03 20:12:49.990521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.990482 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\") pod \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " Nov 03 20:12:49.990892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.990868 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\") pod \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\" (UID: \"75711e96-f618-48d7-b7ad-db6f25b4f4f7\") " Nov 03 20:12:49.991743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.991713 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca" pod "75711e96-f618-48d7-b7ad-db6f25b4f4f7" (UID: "75711e96-f618-48d7-b7ad-db6f25b4f4f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:49.991935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.991905 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42pgj\" (UniqueName: \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.994247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.994223 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.995074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.995052 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.996052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.995912 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:49.996629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:49.996452 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:50.075161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.020258 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca" pod "75711e96-f618-48d7-b7ad-db6f25b4f4f7" (UID: "75711e96-f618-48d7-b7ad-db6f25b4f4f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:50.075161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.020376 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m" pod "75711e96-f618-48d7-b7ad-db6f25b4f4f7" (UID: "75711e96-f618-48d7-b7ad-db6f25b4f4f7"). InnerVolumeSpecName "kube-api-access-hr47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:50.075161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.034545 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:50.075161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.063775 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42pgj\" (UniqueName: \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\") pod \"maas-api-revoke-sub-54a2449b-r2z9j\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:50.086041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.086001 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:50.097561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.097530 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr47m\" (UniqueName: \"kubernetes.io/projected/75711e96-f618-48d7-b7ad-db6f25b4f4f7-kube-api-access-hr47m\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:50.097561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.097560 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75711e96-f618-48d7-b7ad-db6f25b4f4f7-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:50.098073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.098045 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:50.257520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.257441 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.257520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.257501 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:50.388319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.387766 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j"] Nov 03 20:12:50.505468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.505434 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:50.673083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.673053 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" Nov 03 20:12:50.673249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.673090 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p" event={"ID":"75711e96-f618-48d7-b7ad-db6f25b4f4f7","Type":"ContainerDied","Data":"32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714"} Nov 03 20:12:50.673249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.673152 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32317a978b426912bbb9ad0bc07ccdeba89a7151c1bf5f3e4b054e0a19675714" Nov 03 20:12:50.674690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:50.674662 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" event={"ID":"37990633-fb7c-4247-b58a-a63c8749f888","Type":"ContainerStarted","Data":"a8baaab22e016008416564bf79734f912bb30ce239208959e46c11768ed77966"} Nov 03 20:12:51.257409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:51.257374 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.257580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:51.257436 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:51.403973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:51.403942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:51.682465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:51.682435 5954 generic.go:356] "Generic (PLEG): container finished" podID="37990633-fb7c-4247-b58a-a63c8749f888" containerID="4b17145cddc673542c78c04fee409e151e2a4bf0c170c7440a2639d386220980" exitCode=0 Nov 03 20:12:51.682639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:51.682496 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" event={"ID":"37990633-fb7c-4247-b58a-a63c8749f888","Type":"ContainerDied","Data":"4b17145cddc673542c78c04fee409e151e2a4bf0c170c7440a2639d386220980"} Nov 03 20:12:52.257600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.257568 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.258030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.257625 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:52.460394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.460346 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.872718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.872694 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:52.957184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.957080 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\" (UniqueName: \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\") pod \"37990633-fb7c-4247-b58a-a63c8749f888\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " Nov 03 20:12:52.957369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.957243 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\") pod \"37990633-fb7c-4247-b58a-a63c8749f888\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " Nov 03 20:12:52.957369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.957300 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\") pod \"37990633-fb7c-4247-b58a-a63c8749f888\" (UID: \"37990633-fb7c-4247-b58a-a63c8749f888\") " Nov 03 20:12:52.957749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.957691 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca" pod "37990633-fb7c-4247-b58a-a63c8749f888" (UID: "37990633-fb7c-4247-b58a-a63c8749f888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:52.957875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.957864 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:52.960231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.960204 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca" pod "37990633-fb7c-4247-b58a-a63c8749f888" (UID: "37990633-fb7c-4247-b58a-a63c8749f888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:52.960555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:52.960529 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj" pod "37990633-fb7c-4247-b58a-a63c8749f888" (UID: "37990633-fb7c-4247-b58a-a63c8749f888"). InnerVolumeSpecName "kube-api-access-42pgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:53.058710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.058678 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42pgj\" (UniqueName: \"kubernetes.io/projected/37990633-fb7c-4247-b58a-a63c8749f888-kube-api-access-42pgj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:53.058710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.058707 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37990633-fb7c-4247-b58a-a63c8749f888-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:53.257535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.257449 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.257535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.257505 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 20:12:53.390657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.390621 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.459870 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:12:53.459827 5954 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37990633_fb7c_4247_b58a_a63c8749f888.slice/crio-a8baaab22e016008416564bf79734f912bb30ce239208959e46c11768ed77966\": RecentStats: unable to find data in memory cache]" Nov 03 20:12:53.710376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.710336 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" Nov 03 20:12:53.710544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.710338 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j" event={"ID":"37990633-fb7c-4247-b58a-a63c8749f888","Type":"ContainerDied","Data":"a8baaab22e016008416564bf79734f912bb30ce239208959e46c11768ed77966"} Nov 03 20:12:53.710544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:53.710504 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8baaab22e016008416564bf79734f912bb30ce239208959e46c11768ed77966" Nov 03 20:12:54.176577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:54.176538 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:54.271402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:54.271374 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:54.272747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:54.272723 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:54.274737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:54.274716 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:12:55.098832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.098802 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:55.633202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.633171 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:12:55.633959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.633930 5954 cpu_manager.go:414] "containerMap: removing container" podUID="75711e96-f618-48d7-b7ad-db6f25b4f4f7" containerName="revoke-keys" Nov 03 20:12:55.633959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.633959 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75711e96-f618-48d7-b7ad-db6f25b4f4f7" containerName="revoke-keys" Nov 03 20:12:55.634190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.634053 5954 cpu_manager.go:414] "containerMap: removing container" podUID="37990633-fb7c-4247-b58a-a63c8749f888" containerName="revoke-keys" Nov 03 20:12:55.634190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.634063 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37990633-fb7c-4247-b58a-a63c8749f888" containerName="revoke-keys" Nov 03 20:12:55.634307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.634228 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="37990633-fb7c-4247-b58a-a63c8749f888" containerName="revoke-keys" Nov 03 20:12:55.634307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.634243 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="75711e96-f618-48d7-b7ad-db6f25b4f4f7" containerName="revoke-keys" Nov 03 20:12:55.637803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.637782 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.653588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.653561 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:12:55.708424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708394 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708432 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708563 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708593 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708720 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708619 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708720 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708667 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708731 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708806 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.708855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.708846 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hr4\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810326 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hr4\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810522 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810568 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810650 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810684 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810730 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810768 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810797 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810866 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.810883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.810870 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.811089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.811057 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.811246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.811177 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.812073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.812047 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.812229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.812211 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.813449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.813431 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.814289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.814272 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.825646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.825621 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hr4\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.826487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.826470 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:55.875496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.875468 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:55.957458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:55.957008 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:56.250968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.250941 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:12:56.252179 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:56.252155 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac8ca03_d913_4b2e_97f8_c1bd5fa581ac.slice/crio-9ed9f722f745942a004a4d8016d0170b0ddfa25921abca610caebf7b4c5b6783 WatchSource:0}: Error finding container 9ed9f722f745942a004a4d8016d0170b0ddfa25921abca610caebf7b4c5b6783: Status 404 returned error can't find the container with id 9ed9f722f745942a004a4d8016d0170b0ddfa25921abca610caebf7b4c5b6783 Nov 03 20:12:56.254765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.254734 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.254824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.254798 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.254864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.254828 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.713577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.713543 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:56.773176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.773092 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" event={"ID":"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac","Type":"ContainerStarted","Data":"49c6aaa68a56e1565c5268714831a5498270095dde1fa940770a9b5f910dad2c"} Nov 03 20:12:56.773444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.773422 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" event={"ID":"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac","Type":"ContainerStarted","Data":"9ed9f722f745942a004a4d8016d0170b0ddfa25921abca610caebf7b4c5b6783"} Nov 03 20:12:56.803680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.803620 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podStartSLOduration=1.803599634 podStartE2EDuration="1.803599634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:56.802623028 +0000 UTC m=+917.527630665" watchObservedRunningTime="2026-11-03 20:12:56.803599634 +0000 UTC m=+917.528607233" Nov 03 20:12:56.958163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:56.958128 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:12:57.358912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:57.358884 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:57.639489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:57.639392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:57.960048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:57.959969 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:12:57.960048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:57.960020 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:12:58.575210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.575177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:58.848276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.848195 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt"] Nov 03 20:12:58.852407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.852391 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:58.861241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.861217 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:58.861487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.861464 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:58.866044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.866020 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt"] Nov 03 20:12:58.958244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.958209 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:12:58.958415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.958272 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:12:58.964890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.964865 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:58.965208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.964907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:58.965208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:58.964978 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj74\" (UniqueName: \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.066605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.066561 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.066773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.066628 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.066773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.066673 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj74\" (UniqueName: \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.067371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.067351 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.069064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.069046 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.078636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.078611 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj74\" (UniqueName: \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\") pod \"maas-api-revoke-sub-1f331a05-t8lbt\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.168458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.168409 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:12:59.469303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.469219 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:12:59.485557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.485531 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt"] Nov 03 20:12:59.486918 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:12:59.486892 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6722e099_ad7d_4104_a921_d3d19bfdd3e0.slice/crio-73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f WatchSource:0}: Error finding container 73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f: Status 404 returned error can't find the container with id 73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f Nov 03 20:12:59.540437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.540407 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:59.792303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.792226 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" event={"ID":"6722e099-ad7d-4104-a921-d3d19bfdd3e0","Type":"ContainerStarted","Data":"73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f"} Nov 03 20:12:59.958362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.958326 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:12:59.958533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:12:59.958390 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:13:00.427131 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:13:00.426647 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:13:00.806380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:00.806163 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:00.819631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:00.819596 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:13:00.831969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:00.831929 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.011346 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.011403 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.022876 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.022935 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.022963 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.022994 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.023022 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.023283 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.023430 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.023482 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4z8c\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.055132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.023531 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.183796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.160885 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:13:01.183796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.163406 5954 generic.go:356] "Generic (PLEG): container finished" podID="6722e099-ad7d-4104-a921-d3d19bfdd3e0" containerID="03776683a1715db59842471e8742bedd0aa6b439cc4f5300046e240905b50a74" exitCode=0 Nov 03 20:13:01.183796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.163764 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" event={"ID":"6722e099-ad7d-4104-a921-d3d19bfdd3e0","Type":"ContainerDied","Data":"03776683a1715db59842471e8742bedd0aa6b439cc4f5300046e240905b50a74"} Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.449904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450020 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4z8c\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450086 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450369 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450441 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450476 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450515 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.463401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.450561 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.582157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.576062 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.712140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.579477 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.712140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.583706 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.712140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.584252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.712140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.668480 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.712140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.709417 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.745327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.736858 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.958796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.958753 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.972145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.960325 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:01.972145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.960433 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:13:01.979716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.978813 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:01.995313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:01.995264 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:13:02.014726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.014607 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4z8c\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:02.120131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.107846 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.142237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.142194 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:13:02.167390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.167348 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:02.210344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.207205 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:13:02.244514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.235498 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:13:02.244514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.235657 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.262805 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.262869 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7729s\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.262904 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.262946 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.263017 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.263073 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.263314 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.263371 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.268202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.263457 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.380909 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.380976 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381020 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381051 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4v5\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381134 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381169 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381201 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381254 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381293 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381412 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381460 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381516 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381605 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381678 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381708 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381741 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381767 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.382129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.381810 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7729s\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.383224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.382685 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.442351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.403582 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.444540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.443055 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.444540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.443426 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.444540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.444034 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.458945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.458867 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.471181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.462786 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.472369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.472304 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.475333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.475278 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7729s\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.486975 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487039 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487094 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487204 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487345 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487372 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487403 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487450 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.487485 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4v5\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.488566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.488241 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.507230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.507195 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.507989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.507961 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.534191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.508848 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.536911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.536024 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.557775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.557699 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.574391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.572899 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4v5\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.574391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.573260 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.589137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.584964 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.601132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.598326 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:02.864480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.763666 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:02.923714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.923667 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:02.967578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.967543 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:02.967847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:02.967818 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:13:03.263424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.263395 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:13:03.284173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.284143 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\") pod \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " Nov 03 20:13:03.284826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.284802 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\") pod \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " Nov 03 20:13:03.285439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.285422 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\" (UniqueName: \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\") pod \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\" (UID: \"6722e099-ad7d-4104-a921-d3d19bfdd3e0\") " Nov 03 20:13:03.286394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.284692 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca" pod "6722e099-ad7d-4104-a921-d3d19bfdd3e0" (UID: "6722e099-ad7d-4104-a921-d3d19bfdd3e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:03.289883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.289854 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca" pod "6722e099-ad7d-4104-a921-d3d19bfdd3e0" (UID: "6722e099-ad7d-4104-a921-d3d19bfdd3e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:03.291579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.291552 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74" pod "6722e099-ad7d-4104-a921-d3d19bfdd3e0" (UID: "6722e099-ad7d-4104-a921-d3d19bfdd3e0"). InnerVolumeSpecName "kube-api-access-drj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:03.348230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.348203 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" Nov 03 20:13:03.348403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.348246 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt" event={"ID":"6722e099-ad7d-4104-a921-d3d19bfdd3e0","Type":"ContainerDied","Data":"73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f"} Nov 03 20:13:03.348403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.348310 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73fbb1e887b40f184527c05224cd32995c732557601e529ed6525717b1f2dd9f" Nov 03 20:13:03.366695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.366670 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:13:03.371224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.371194 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.371325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.371259 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.371325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.371287 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.387149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.387118 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:13:03.387575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.387550 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj74\" (UniqueName: \"kubernetes.io/projected/6722e099-ad7d-4104-a921-d3d19bfdd3e0-kube-api-access-drj74\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:03.387656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.387576 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:03.387656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.387587 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6722e099-ad7d-4104-a921-d3d19bfdd3e0-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:03.388535 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:03.388512 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c02de98_e73f_4542_b221_d088eff28d0d.slice/crio-40e03e7945e96f62631bb62e2cbb1f83034833d83c98a7fd9b2026f5a7c30dc0 WatchSource:0}: Error finding container 40e03e7945e96f62631bb62e2cbb1f83034833d83c98a7fd9b2026f5a7c30dc0: Status 404 returned error can't find the container with id 40e03e7945e96f62631bb62e2cbb1f83034833d83c98a7fd9b2026f5a7c30dc0 Nov 03 20:13:03.391161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.391084 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.391261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.391202 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.391261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.391249 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.447021 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:03.446991 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fb5d90_f035_41ea_b267_3434641359d4.slice/crio-53e74269a694993e1399b4dca88bbc63d12dbe6dd9ea7f87034321c1e854dc8d WatchSource:0}: Error finding container 53e74269a694993e1399b4dca88bbc63d12dbe6dd9ea7f87034321c1e854dc8d: Status 404 returned error can't find the container with id 53e74269a694993e1399b4dca88bbc63d12dbe6dd9ea7f87034321c1e854dc8d Nov 03 20:13:03.447246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.447222 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:13:03.449859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.449829 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.449942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.449894 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.449984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.449941 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:03.643129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.634442 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:03.799739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.799592 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:03.972332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.971793 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:13:03.972332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:03.971856 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:13:04.409163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.402994 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" event={"ID":"fea58ac0-9d23-4327-b902-3bd340ebd7f6","Type":"ContainerStarted","Data":"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960"} Nov 03 20:13:04.409163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.403082 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" event={"ID":"fea58ac0-9d23-4327-b902-3bd340ebd7f6","Type":"ContainerStarted","Data":"4521fb28798601af225c7888ffcc5c50f0d3307c7402bb54c23be69ac2d8c48e"} Nov 03 20:13:04.409163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.406932 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" event={"ID":"e2fb5d90-f035-41ea-b267-3434641359d4","Type":"ContainerStarted","Data":"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994"} Nov 03 20:13:04.409163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.407011 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" event={"ID":"e2fb5d90-f035-41ea-b267-3434641359d4","Type":"ContainerStarted","Data":"53e74269a694993e1399b4dca88bbc63d12dbe6dd9ea7f87034321c1e854dc8d"} Nov 03 20:13:04.412128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.411617 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" event={"ID":"3c02de98-e73f-4542-b221-d088eff28d0d","Type":"ContainerStarted","Data":"895250e8b10996f794666540583eb6a6e7a8de7dc95c6eb7b6379c2d22c44836"} Nov 03 20:13:04.412128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.411677 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" event={"ID":"3c02de98-e73f-4542-b221-d088eff28d0d","Type":"ContainerStarted","Data":"40e03e7945e96f62631bb62e2cbb1f83034833d83c98a7fd9b2026f5a7c30dc0"} Nov 03 20:13:04.460131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.459498 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podStartSLOduration=4.459472837 podStartE2EDuration="4.459472837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:04.435455495 +0000 UTC m=+925.160463092" watchObservedRunningTime="2026-11-03 20:13:04.459472837 +0000 UTC m=+925.184480434" Nov 03 20:13:04.512624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.512563 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podStartSLOduration=3.512542373 podStartE2EDuration="3.512542373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:04.510398918 +0000 UTC m=+925.235406516" watchObservedRunningTime="2026-11-03 20:13:04.512542373 +0000 UTC m=+925.237549970" Nov 03 20:13:04.599399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.599345 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:04.603114 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.602755 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:04.604575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.604532 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:04.620131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.606542 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podStartSLOduration=3.606521022 podStartE2EDuration="3.606521022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:04.558829369 +0000 UTC m=+925.283837004" watchObservedRunningTime="2026-11-03 20:13:04.606521022 +0000 UTC m=+925.331528619" Nov 03 20:13:04.644524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.606840 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:04.777154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.777048 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:04.807651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.807608 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:04.812400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:04.812359 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:05.204295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.204257 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:05.205092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.205065 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:05.206836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.205367 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:05.505181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.503514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:05.599724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.599684 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:05.599958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.599932 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:05.769017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.768926 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:05.769618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.769584 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:05.983736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.983698 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:05.984008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:05.983980 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:06.181551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.181514 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:06.181871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.181827 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:06.382133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.380992 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:06.602338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.600526 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:06.602338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.600586 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:06.767134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.766122 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:06.767134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.766366 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:06.999187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.998226 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:06.999187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:06.998319 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:07.011136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.008319 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:13:07.011136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.010048 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:13:07.011611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.011586 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:13:07.101133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.098636 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:07.178132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.177848 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:07.178132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.177912 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:07.599058 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.599016 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:07.599309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.599082 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:07.783127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.782522 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:07.783127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.782588 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:07.893229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:07.889649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:08.203224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.186050 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:08.203224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.186128 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:08.599585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.599497 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:08.599585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.599558 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:08.702033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.702001 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:08.781619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.781587 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:08.781955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.781914 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:08.818150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:08.810798 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:09.183810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.183773 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:09.184238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.184210 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:09.397140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.396816 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:09.602197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.599629 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:09.602197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.599714 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:09.771133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.770282 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:09.771133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:09.770363 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:10.090187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.089752 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:10.183129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.176677 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:10.183129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.176743 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:10.379132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.376427 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:10.608126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.607965 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:13:10.608126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.608045 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 20:13:10.764848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.764732 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:10.764848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:10.764818 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:11.177162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:11.176809 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:13:11.177162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:11.176872 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 20:13:11.204244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:11.204180 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:11.769841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:11.769805 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:11.771243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:11.771184 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 20:13:12.605433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:12.605384 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:12.605760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:12.605710 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:12.693083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:12.693043 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:13.193815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.193206 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:13.193815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.193272 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:13.522537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.522446 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:13.543545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.543514 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:13.586177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.576391 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:13:13.634603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.634277 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:13.649757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.649652 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:13.650613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.650219 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:13.650613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:13.650311 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:14.122505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.122468 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm"] Nov 03 20:13:14.123659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.123629 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6722e099-ad7d-4104-a921-d3d19bfdd3e0" containerName="revoke-keys" Nov 03 20:13:14.123845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.123827 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6722e099-ad7d-4104-a921-d3d19bfdd3e0" containerName="revoke-keys" Nov 03 20:13:14.124212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.124191 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6722e099-ad7d-4104-a921-d3d19bfdd3e0" containerName="revoke-keys" Nov 03 20:13:14.192655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.192613 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.206264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.206232 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:14.206840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.206309 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:13:14.209532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.209500 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:14.209662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.209556 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:14.212774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.211081 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm"] Nov 03 20:13:14.303220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.303176 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.303530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.303510 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9cd\" (UniqueName: \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.303680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.303664 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.313617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.313590 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:14.314770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.314747 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:14.317348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.317326 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:13:14.406060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.405959 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.406281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.405273 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.406820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.406454 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9cd\" (UniqueName: \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.408207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.408177 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.435870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.435812 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.465442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.465411 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9cd\" (UniqueName: \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\") pod \"maas-api-revoke-sub-52d6db61-2vdrm\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.516163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.516128 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:14.517062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.517031 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:14.518474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.518199 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:13:14.595199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.593564 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:14.724156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.721555 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:14.924333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:14.924025 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm"] Nov 03 20:13:15.084048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.084012 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" event={"ID":"f570a69f-e77c-456a-8954-b578aebf950c","Type":"ContainerStarted","Data":"3e79d12269ab3ef84a07e49d2b635f58610e901c2dbd96512e03f3294ce1a69e"} Nov 03 20:13:15.519565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.519485 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:15.795690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.795611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:15.931132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.930412 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:13:15.948055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.947162 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:15.975175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.974705 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:13:15.981199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:15.981166 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:13:16.018604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.011639 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.038167 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.039714 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.039765 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.039848 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.039921 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.039995 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.040154 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.040228 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2zc\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.040308 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.040687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.040546 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.092324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.092291 5954 generic.go:356] "Generic (PLEG): container finished" podID="f570a69f-e77c-456a-8954-b578aebf950c" containerID="f0bf4dc71f27ae278234f13bd8d61d21cc0b631754fa703319fdc8f306b70c71" exitCode=0 Nov 03 20:13:16.092515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.092373 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" event={"ID":"f570a69f-e77c-456a-8954-b578aebf950c","Type":"ContainerDied","Data":"f0bf4dc71f27ae278234f13bd8d61d21cc0b631754fa703319fdc8f306b70c71"} Nov 03 20:13:16.141756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.141725 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.141756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.141760 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.141890 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.141946 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2zc\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142024 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142165 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142219 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142263 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142327 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142359 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142388 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142406 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142358 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142480 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142570 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142572 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142628 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142656 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142658 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142757 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.142902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142813 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8jp\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.142902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.142854 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.143234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.143214 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.145145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.145121 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.145260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.145183 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.154079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.154057 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.154267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.154206 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2zc\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.245076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.244731 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.244794 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.244842 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.244952 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.244987 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245146 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8jp\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245272 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245295 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245341 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245386 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245400 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245459 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245548 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.245775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.245766 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.247410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.247388 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.247657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.247639 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.263695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.263661 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.263816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.263757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8jp\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.347500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.347410 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:16.360081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.360050 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:16.644198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.644166 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:13:16.644997 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:16.644971 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadcc3b3_377d_4478_b91c_c2957577306a.slice/crio-197a36a20d0b1bff3892481a674c7e0af77f667adc71e335e0a829166151f24e WatchSource:0}: Error finding container 197a36a20d0b1bff3892481a674c7e0af77f667adc71e335e0a829166151f24e: Status 404 returned error can't find the container with id 197a36a20d0b1bff3892481a674c7e0af77f667adc71e335e0a829166151f24e Nov 03 20:13:16.647992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.647962 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.648077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.648022 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.648077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.648058 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.684466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.684443 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:13:16.691400 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:16.691368 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ab4577_de9d_4a19_8e00_9f546754719c.slice/crio-1f13a6d7cc9cd9676be78ad612acf36acbf3f6b00fefd6d5e85e1b6ae9e901eb WatchSource:0}: Error finding container 1f13a6d7cc9cd9676be78ad612acf36acbf3f6b00fefd6d5e85e1b6ae9e901eb: Status 404 returned error can't find the container with id 1f13a6d7cc9cd9676be78ad612acf36acbf3f6b00fefd6d5e85e1b6ae9e901eb Nov 03 20:13:16.694168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.694136 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.694272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.694217 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.694272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.694264 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.794283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:16.794247 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:17.103197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.103137 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" event={"ID":"b8ab4577-de9d-4a19-8e00-9f546754719c","Type":"ContainerStarted","Data":"7037319c30b55726056e2cf42933f70025ecf8ab55faeb047fc28af6fe113947"} Nov 03 20:13:17.103403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.103219 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" event={"ID":"b8ab4577-de9d-4a19-8e00-9f546754719c","Type":"ContainerStarted","Data":"1f13a6d7cc9cd9676be78ad612acf36acbf3f6b00fefd6d5e85e1b6ae9e901eb"} Nov 03 20:13:17.104940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.104904 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" event={"ID":"cadcc3b3-377d-4478-b91c-c2957577306a","Type":"ContainerStarted","Data":"bf1dfad88173ffec6d0e94577ef1616b010f1fc5ab7d814cf559b52320a4f0f7"} Nov 03 20:13:17.105080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.104962 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" event={"ID":"cadcc3b3-377d-4478-b91c-c2957577306a","Type":"ContainerStarted","Data":"197a36a20d0b1bff3892481a674c7e0af77f667adc71e335e0a829166151f24e"} Nov 03 20:13:17.142991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.142924 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podStartSLOduration=2.142905819 podStartE2EDuration="2.142905819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.140195862 +0000 UTC m=+937.865203459" watchObservedRunningTime="2026-11-03 20:13:17.142905819 +0000 UTC m=+937.867913424" Nov 03 20:13:17.174975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.174924 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podStartSLOduration=2.174907321 podStartE2EDuration="2.174907321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.167554897 +0000 UTC m=+937.892562493" watchObservedRunningTime="2026-11-03 20:13:17.174907321 +0000 UTC m=+937.899914917" Nov 03 20:13:17.245426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.245392 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:17.347939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.347903 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:17.349547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.349520 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:17.349672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.349576 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:17.360767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.360688 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:17.362436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.362397 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:17.362580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.362458 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:17.367712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.366504 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\" (UniqueName: \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\") pod \"f570a69f-e77c-456a-8954-b578aebf950c\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " Nov 03 20:13:17.367712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.366605 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\") pod \"f570a69f-e77c-456a-8954-b578aebf950c\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " Nov 03 20:13:17.367712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.366751 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\") pod \"f570a69f-e77c-456a-8954-b578aebf950c\" (UID: \"f570a69f-e77c-456a-8954-b578aebf950c\") " Nov 03 20:13:17.368265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.368236 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca" pod "f570a69f-e77c-456a-8954-b578aebf950c" (UID: "f570a69f-e77c-456a-8954-b578aebf950c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:17.373511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.373478 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca" pod "f570a69f-e77c-456a-8954-b578aebf950c" (UID: "f570a69f-e77c-456a-8954-b578aebf950c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:17.374201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.374142 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd" pod "f570a69f-e77c-456a-8954-b578aebf950c" (UID: "f570a69f-e77c-456a-8954-b578aebf950c"). InnerVolumeSpecName "kube-api-access-tq9cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:17.473138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.469402 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:17.473138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.469443 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9cd\" (UniqueName: \"kubernetes.io/projected/f570a69f-e77c-456a-8954-b578aebf950c-kube-api-access-tq9cd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:17.473138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.469462 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f570a69f-e77c-456a-8954-b578aebf950c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:17.955132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:17.954638 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:18.196868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.196832 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" event={"ID":"f570a69f-e77c-456a-8954-b578aebf950c","Type":"ContainerDied","Data":"3e79d12269ab3ef84a07e49d2b635f58610e901c2dbd96512e03f3294ce1a69e"} Nov 03 20:13:18.197188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.196935 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm" Nov 03 20:13:18.197315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.197300 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e79d12269ab3ef84a07e49d2b635f58610e901c2dbd96512e03f3294ce1a69e" Nov 03 20:13:18.349695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.349597 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.349959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.349933 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:18.360691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.360653 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.360964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.360937 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:18.772131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.771841 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:13:18.773508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.772879 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f570a69f-e77c-456a-8954-b578aebf950c" containerName="revoke-keys" Nov 03 20:13:18.773508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.772904 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f570a69f-e77c-456a-8954-b578aebf950c" containerName="revoke-keys" Nov 03 20:13:18.773508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.773123 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f570a69f-e77c-456a-8954-b578aebf950c" containerName="revoke-keys" Nov 03 20:13:18.777583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.777543 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.780672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.780538 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:13:18.782299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.782277 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"distinct-w0-b1b190-kserve-self-signed-certs\"" Nov 03 20:13:18.840130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.825944 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:18.919134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.916077 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.952131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.951333 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.952131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.951426 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.952131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.951584 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.952131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.951617 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:18.952131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:18.951677 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054091 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054223 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054278 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054410 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054435 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.054477 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.055849 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.056385 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.058315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.056449 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.076130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.069685 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.097129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.087291 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.101131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.099577 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\") pod \"distinct-w0-b1b190-kserve-c75c78b9b-zzjrv\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.364134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.358739 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.364134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.358789 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:19.364134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.363353 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.364134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.363402 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:19.394297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.394262 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:19.743747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.743597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:19.800466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:19.800429 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:13:20.212878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.212841 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" event={"ID":"15ac93b8-880b-4a16-bc42-ab30525a26af","Type":"ContainerStarted","Data":"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf"} Nov 03 20:13:20.212878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.212888 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" event={"ID":"15ac93b8-880b-4a16-bc42-ab30525a26af","Type":"ContainerStarted","Data":"e8f2697cbed78887f9dc6b48c544b0297cc3a320c370e01d57934e946d4b874e"} Nov 03 20:13:20.242131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.233541 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podStartSLOduration=2.233524526 podStartE2EDuration="2.233524526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:20.232716723 +0000 UTC m=+940.957724320" watchObservedRunningTime="2026-11-03 20:13:20.233524526 +0000 UTC m=+940.958532126" Nov 03 20:13:20.350188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.349531 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.350188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.349595 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:20.364132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.363717 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.364132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.363776 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:20.399133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:20.398675 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:21.190278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.190235 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:21.208215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.208181 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x"] Nov 03 20:13:21.212680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.212656 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.320195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.264314 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:21.320195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.264456 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:13:21.405154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.404685 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.405154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.404751 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:21.481981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.407926 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.481981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.408028 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.481981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.408116 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbdn\" (UniqueName: \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.481981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.408645 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.481981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.408718 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:21.490494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.490456 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x"] Nov 03 20:13:21.511236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.511170 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbdn\" (UniqueName: \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.583475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.583435 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.585664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.585633 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.586753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.586730 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbdn\" (UniqueName: \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.592271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.592248 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.596491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.596468 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0558c9f6-tfn7x\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:21.785668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:21.785582 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:22.161132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.139080 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x"] Nov 03 20:13:22.362194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.354961 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.362194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.355025 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:22.364124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.364080 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.364324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.364301 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:22.411614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.411572 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" event={"ID":"ed3bd205-2d06-4df7-a578-19df49be73d4","Type":"ContainerStarted","Data":"dadcda6427f853e3ac1b6a8bae3bf60a914c1d141d13b6a77e9901a9bec8bbb2"} Nov 03 20:13:22.769882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:22.769840 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:23.348427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.348394 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.348557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.348462 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 20:13:23.361801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.361736 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.361952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.361862 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 20:13:23.422632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.422592 5954 generic.go:356] "Generic (PLEG): container finished" podID="ed3bd205-2d06-4df7-a578-19df49be73d4" containerID="2081314dfb3b493f3cd2970a8f24dc81b13351f89dbb2a34fdf79bee159e3e7d" exitCode=0 Nov 03 20:13:23.423145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:23.422706 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" event={"ID":"ed3bd205-2d06-4df7-a578-19df49be73d4","Type":"ContainerDied","Data":"2081314dfb3b493f3cd2970a8f24dc81b13351f89dbb2a34fdf79bee159e3e7d"} Nov 03 20:13:24.014817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.014770 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:24.354384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.354358 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:24.355233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.355206 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:24.356036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.356019 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:13:24.429623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.429593 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:24.431974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.431951 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:24.433933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.433911 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:13:24.652049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.651995 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:24.758292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.758252 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\") pod \"ed3bd205-2d06-4df7-a578-19df49be73d4\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " Nov 03 20:13:24.758471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.758327 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\") pod \"ed3bd205-2d06-4df7-a578-19df49be73d4\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " Nov 03 20:13:24.758471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.758462 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\" (UniqueName: \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\") pod \"ed3bd205-2d06-4df7-a578-19df49be73d4\" (UID: \"ed3bd205-2d06-4df7-a578-19df49be73d4\") " Nov 03 20:13:24.759047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.758944 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca" pod "ed3bd205-2d06-4df7-a578-19df49be73d4" (UID: "ed3bd205-2d06-4df7-a578-19df49be73d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:24.761226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.761201 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca" pod "ed3bd205-2d06-4df7-a578-19df49be73d4" (UID: "ed3bd205-2d06-4df7-a578-19df49be73d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:24.761322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.761240 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn" pod "ed3bd205-2d06-4df7-a578-19df49be73d4" (UID: "ed3bd205-2d06-4df7-a578-19df49be73d4"). InnerVolumeSpecName "kube-api-access-szbdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:24.860778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.860694 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:24.860778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.860729 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3bd205-2d06-4df7-a578-19df49be73d4-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:24.860778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:24.860747 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbdn\" (UniqueName: \"kubernetes.io/projected/ed3bd205-2d06-4df7-a578-19df49be73d4-kube-api-access-szbdn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:25.448643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:25.448604 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" Nov 03 20:13:25.449095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:25.448648 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x" event={"ID":"ed3bd205-2d06-4df7-a578-19df49be73d4","Type":"ContainerDied","Data":"dadcda6427f853e3ac1b6a8bae3bf60a914c1d141d13b6a77e9901a9bec8bbb2"} Nov 03 20:13:25.449095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:25.448723 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dadcda6427f853e3ac1b6a8bae3bf60a914c1d141d13b6a77e9901a9bec8bbb2" Nov 03 20:13:25.496655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:25.496613 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:26.553711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:26.553673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:27.121741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.121705 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:13:27.122672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.122650 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ed3bd205-2d06-4df7-a578-19df49be73d4" containerName="revoke-keys" Nov 03 20:13:27.122730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.122676 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3bd205-2d06-4df7-a578-19df49be73d4" containerName="revoke-keys" Nov 03 20:13:27.122915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.122900 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3bd205-2d06-4df7-a578-19df49be73d4" containerName="revoke-keys" Nov 03 20:13:27.127139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.127096 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.138015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.137453 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"kube-root-ca.crt\"" Nov 03 20:13:27.138015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.138002 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"openshift-service-ca.crt\"" Nov 03 20:13:27.139102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.139073 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"distinct-2-w4-3aba31-kserve-self-signed-certs\"" Nov 03 20:13:27.139240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.139142 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"default-dockercfg-rcqfz\"" Nov 03 20:13:27.140344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.140322 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:13:27.185884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.185843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:27.199291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199258 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.199467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199300 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.199467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199412 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.199467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199441 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.199637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199505 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsps9\" (UniqueName: \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.199637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.199592 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303263 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303793 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303857 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303898 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303976 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsps9\" (UniqueName: \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.304126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.304137 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.310129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.304593 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.310129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.303701 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.310129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.304852 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.329745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.329375 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.329745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.329487 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsps9\" (UniqueName: \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.356132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.354771 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\") pod \"distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.441750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.441665 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:27.593998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.593951 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:27.731424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:27.731391 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:13:27.732039 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:27.732007 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9efdcc23_c3fc_447b_92cb_a9decf3ece60.slice/crio-c9867f804a43cc0ca3ac61702acd80d193211a9c8e3a1cfe1e6ca45b1134ba42 WatchSource:0}: Error finding container c9867f804a43cc0ca3ac61702acd80d193211a9c8e3a1cfe1e6ca45b1134ba42: Status 404 returned error can't find the container with id c9867f804a43cc0ca3ac61702acd80d193211a9c8e3a1cfe1e6ca45b1134ba42 Nov 03 20:13:28.428423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:28.428388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:28.471473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:28.471439 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" event={"ID":"9efdcc23-c3fc-447b-92cb-a9decf3ece60","Type":"ContainerStarted","Data":"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41"} Nov 03 20:13:28.471723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:28.471701 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" event={"ID":"9efdcc23-c3fc-447b-92cb-a9decf3ece60","Type":"ContainerStarted","Data":"c9867f804a43cc0ca3ac61702acd80d193211a9c8e3a1cfe1e6ca45b1134ba42"} Nov 03 20:13:28.514140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:28.512057 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podStartSLOduration=1.511869845 podStartE2EDuration="1.511869845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:28.509535909 +0000 UTC m=+949.234543526" watchObservedRunningTime="2026-11-03 20:13:28.511869845 +0000 UTC m=+949.236877483" Nov 03 20:13:29.172205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.172172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:29.394854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.394820 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:29.408967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.408939 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:29.442099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.442014 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:29.498879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.498853 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:29.972889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:29.972844 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:30.258905 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:13:30.258824 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:13:30.701793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.701762 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:30.939527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.939494 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:13:30.943787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.943768 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:30.957878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.957808 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:13:30.995757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.995730 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:13:30.999976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:30.999958 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.015162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015129 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015417 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015460 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015575 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015610 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015735 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015771 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015833 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.015965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.015940 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.016618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.016597 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:13:31.030658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.030626 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:13:31.037855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.037584 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.046614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.046573 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:13:31.117291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117261 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117304 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.117478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117331 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117351 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.117598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117497 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117527 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.117598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117571 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117608 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117672 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117694 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117717 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.117856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117785 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d89t\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.117856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117835 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rhgf\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.117913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117869 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.117913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117879 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.117913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.117900 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118005 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118045 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118129 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118234 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.118303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118286 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.118352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118338 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.118404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118386 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118415 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.118473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118437 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.118547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118535 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.118581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118565 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.118623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118587 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.118678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118618 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.118772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118755 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.118835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118785 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.118889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.118862 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.120709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.120684 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.120826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.120751 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.136147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.136091 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.136707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.136676 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.220965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.220861 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.220965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.220918 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.220965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.220958 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221021 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221050 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d89t\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221080 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rhgf\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221133 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221175 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221218 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221247 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221277 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221398 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221444 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221472 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221513 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221547 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221585 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.221699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.221611 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.222098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.222076 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.222400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.222381 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.223330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.223022 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.225159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.224393 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.225159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.224737 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.225159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.224987 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.225159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.225090 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.225442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.225222 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.225442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.225250 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.225442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.225411 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.225669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.225637 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.226310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.226285 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.228544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.228520 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.230522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.230496 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.240097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.240071 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rhgf\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.240886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.240858 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.241055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.241019 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.241346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.241265 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d89t\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.260397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.260370 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:31.320386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.320358 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:31.372101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.371867 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:31.550398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.550368 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:13:31.550542 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:31.550480 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7548a_59dd_4961_aa1f_94a4789b890a.slice/crio-b916367e3c53c0a3d9dbc33695c05eedcb7193e439f994ee270b7fd465b5f2d1 WatchSource:0}: Error finding container b916367e3c53c0a3d9dbc33695c05eedcb7193e439f994ee270b7fd465b5f2d1: Status 404 returned error can't find the container with id b916367e3c53c0a3d9dbc33695c05eedcb7193e439f994ee270b7fd465b5f2d1 Nov 03 20:13:31.552996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.552965 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.553087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.553030 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.553087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.553059 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.627011 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:31.626973 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc68032_d375_4667_84b7_59389b5fb9ee.slice/crio-54f8fc005de91df98a65326707ca2a0ba978d821d5273e7c9db9608ea88ca76a WatchSource:0}: Error finding container 54f8fc005de91df98a65326707ca2a0ba978d821d5273e7c9db9608ea88ca76a: Status 404 returned error can't find the container with id 54f8fc005de91df98a65326707ca2a0ba978d821d5273e7c9db9608ea88ca76a Nov 03 20:13:31.627392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.627368 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:13:31.629514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.629484 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.629624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.629539 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.629624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.629569 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.686053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.686021 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:13:31.690668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.690619 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.690761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.690731 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.690811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.690784 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.720282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:31.720254 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:32.517303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.514504 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" event={"ID":"dc59581e-6842-470a-a017-335297a600e7","Type":"ContainerStarted","Data":"cc1de6d805a8fbdcdcf253c45d5cb94f7f40499ed6539c64269a7f00dae68cbc"} Nov 03 20:13:32.517303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.514605 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" event={"ID":"dc59581e-6842-470a-a017-335297a600e7","Type":"ContainerStarted","Data":"21ffea51c614cc6afb08ab9fe79820d6c9ec8ae248126f7361eb20fdaebe6c05"} Nov 03 20:13:32.540500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.540458 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" event={"ID":"46f7548a-59dd-4961-aa1f-94a4789b890a","Type":"ContainerStarted","Data":"2d91796ec428550701beab6fbd3059670d1f5a3b497d02cf8dc5fa841c2cc58e"} Nov 03 20:13:32.540717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.540532 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" event={"ID":"46f7548a-59dd-4961-aa1f-94a4789b890a","Type":"ContainerStarted","Data":"b916367e3c53c0a3d9dbc33695c05eedcb7193e439f994ee270b7fd465b5f2d1"} Nov 03 20:13:32.542606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.542579 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" event={"ID":"0bc68032-d375-4667-84b7-59389b5fb9ee","Type":"ContainerStarted","Data":"a03acdb717b31d3131d6ca70386b8445aeacf0a03926c069dad89269516e55cb"} Nov 03 20:13:32.542759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.542628 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" event={"ID":"0bc68032-d375-4667-84b7-59389b5fb9ee","Type":"ContainerStarted","Data":"54f8fc005de91df98a65326707ca2a0ba978d821d5273e7c9db9608ea88ca76a"} Nov 03 20:13:32.558419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.558365 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podStartSLOduration=2.558350993 podStartE2EDuration="2.558350993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.55585743 +0000 UTC m=+953.280865025" watchObservedRunningTime="2026-11-03 20:13:32.558350993 +0000 UTC m=+953.283358588" Nov 03 20:13:32.581441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.580022 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podStartSLOduration=2.580001893 podStartE2EDuration="2.580001893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.579143491 +0000 UTC m=+953.304151092" watchObservedRunningTime="2026-11-03 20:13:32.580001893 +0000 UTC m=+953.305009500" Nov 03 20:13:32.608766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.605011 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podStartSLOduration=2.604993399 podStartE2EDuration="2.604993399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.60461067 +0000 UTC m=+953.329618267" watchObservedRunningTime="2026-11-03 20:13:32.604993399 +0000 UTC m=+953.330000997" Nov 03 20:13:32.794216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.787537 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:13:32.794216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:32.789183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:33.267129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.264483 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:33.272132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.268236 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.272132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.268309 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:33.328211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.321281 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:33.328211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.326239 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.328211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.326303 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:33.383131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.373810 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:33.383131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.373998 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.383131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:33.374047 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:34.267336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.267298 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.275059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.275016 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:34.328990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.328528 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.328990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.328589 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:34.397129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.389481 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.397129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.389544 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:34.453129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:34.446678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:35.262134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.261522 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.262134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.261586 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:35.342129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.336583 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.342129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.336660 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:35.376588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.375513 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.376588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.375578 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:35.785131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.781742 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.812489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.812438 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.872215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.818157 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:13:36.020324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.882581 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:36.020324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:35.927203 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:13:36.496133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.474733 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.522456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.522407 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:36.592156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.493091 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.713132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.704059 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.713132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.704152 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:36.713132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.705325 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"distinct-2-w3-62221d-kserve-self-signed-certs\"" Nov 03 20:13:36.789193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.783563 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.789193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.783648 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:36.834153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.833350 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:13:36.843546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.843233 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.856177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.848008 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:36.968587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.867978 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.945064 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.978426 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmg8\" (UniqueName: \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.978524 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.978598 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.978654 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.978789 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.979360 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:13:36.983188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:36.981958 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.001274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.000648 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.097546 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.097718 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.097924 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.098147 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmg8\" (UniqueName: \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.098232 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.098308 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.099001 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.099407 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.104701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.104283 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.510354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.510252 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.554350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.553668 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.554350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.553764 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.554350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.553813 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:37.554350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.554050 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.554350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.554082 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:37.555358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.555333 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.558468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.558441 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:37.580892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.560344 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.665131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.615059 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmg8\" (UniqueName: \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\") pod \"distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:37.876134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.859929 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.919139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.885944 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:38.042317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.978614 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.042317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.980820 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:38.042317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:37.982223 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.065170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.051289 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:38.065170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.051359 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.276806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.276769 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.277081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.277049 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:38.346147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.324523 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.346147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.324588 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:38.433422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.429814 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.437439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.437403 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:38.558192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.534218 5954 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.800139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.778471 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:13:38.893165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.876928 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.907416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.907371 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:38.984850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:38.984782 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.273306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.273197 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.273546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.273519 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:39.298836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.298807 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:13:39.308266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.307415 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:39.406124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.406072 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.406406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.406375 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:39.411378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.411350 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.411567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:39.411543 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:40.269131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.252531 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:13:40.310353 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:40.307696 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb619c24_a539_4182_8f1c_2c72af9f106e.slice/crio-15def84ad78a1c3ae7802529012f4244182ef7e8bf8fdd8d65c700c377437c7f WatchSource:0}: Error finding container 15def84ad78a1c3ae7802529012f4244182ef7e8bf8fdd8d65c700c377437c7f: Status 404 returned error can't find the container with id 15def84ad78a1c3ae7802529012f4244182ef7e8bf8fdd8d65c700c377437c7f Nov 03 20:13:40.311986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.311952 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.312242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.312214 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:40.391139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.339093 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.391139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.339165 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:40.394780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.394749 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.395009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.394966 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:40.406931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:40.406907 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" event={"ID":"cb619c24-a539-4182-8f1c-2c72af9f106e","Type":"ContainerStarted","Data":"15def84ad78a1c3ae7802529012f4244182ef7e8bf8fdd8d65c700c377437c7f"} Nov 03 20:13:41.012005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.011102 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:41.264190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.263985 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.264190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.264044 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:41.335583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.335545 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.336198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.336167 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:41.377133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.376556 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.377133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.376630 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:41.447135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.443297 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" event={"ID":"cb619c24-a539-4182-8f1c-2c72af9f106e","Type":"ContainerStarted","Data":"73c946123f596bc156e83b4c84d7c4434c036e75364c19251dadcc22abc1d6ca"} Nov 03 20:13:41.486168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.479781 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podStartSLOduration=6.479761311 podStartE2EDuration="6.479761311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:41.477939715 +0000 UTC m=+962.202947312" watchObservedRunningTime="2026-11-03 20:13:41.479761311 +0000 UTC m=+962.204768910" Nov 03 20:13:41.895400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:41.895362 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:42.200827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.200680 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:42.264152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.260887 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.264152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.260946 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 20:13:42.321412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.321365 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.321574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.321439 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 20:13:42.373302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.373262 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.373680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:42.373332 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:13:43.288991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:43.288949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:43.379164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:43.379099 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:43.379959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:43.379937 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:43.380784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:43.380766 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:13:44.190729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.190690 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:44.218828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.218278 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:44.219492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.219353 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:44.220379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.220345 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:13:44.371554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.365203 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:44.371554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:44.365266 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:45.002409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:45.002378 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:45.003203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:45.003029 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:45.004138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:45.004085 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:13:45.133399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:45.133367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:46.091351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:46.091318 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.014842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.014809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.305663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.305584 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:13:47.311009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.310981 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.318967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.318937 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"distinct-w2-90ef9c-kserve-self-signed-certs\"" Nov 03 20:13:47.319797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.319771 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:13:47.321553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.321531 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"default-dockercfg-db8p9\"" Nov 03 20:13:47.324013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.323994 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"openshift-service-ca.crt\"" Nov 03 20:13:47.324148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.324129 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"kube-root-ca.crt\"" Nov 03 20:13:47.351486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.351457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.413348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413307 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.413510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413477 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.413624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413606 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx767\" (UniqueName: \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.413672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413654 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.413781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413770 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.413836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.413798 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.481135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.480425 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518032 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx767\" (UniqueName: \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518088 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518250 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518286 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518319 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518436 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.518987 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.522145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.520021 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.529128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.525436 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.529128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.525772 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.563175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.562426 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx767\" (UniqueName: \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.565752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.565724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\") pod \"distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.629630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.629356 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:47.664595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.664569 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:47.679372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.679341 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:47.896031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.896004 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:47.905453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.905423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.908859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.908840 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:47.938498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:47.938475 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:13:47.939012 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:47.938988 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31bec522_d1ff_47fc_b8df_b862eb1faae5.slice/crio-3b2bbd3e8be1ff3aa5aeb85447d2eec9a1e63d94d40097336009718d7392bbd4 WatchSource:0}: Error finding container 3b2bbd3e8be1ff3aa5aeb85447d2eec9a1e63d94d40097336009718d7392bbd4: Status 404 returned error can't find the container with id 3b2bbd3e8be1ff3aa5aeb85447d2eec9a1e63d94d40097336009718d7392bbd4 Nov 03 20:13:48.670134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:48.670085 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" event={"ID":"31bec522-d1ff-47fc-b8df-b862eb1faae5","Type":"ContainerStarted","Data":"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549"} Nov 03 20:13:48.670633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:48.670147 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" event={"ID":"31bec522-d1ff-47fc-b8df-b862eb1faae5","Type":"ContainerStarted","Data":"3b2bbd3e8be1ff3aa5aeb85447d2eec9a1e63d94d40097336009718d7392bbd4"} Nov 03 20:13:48.682726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:48.682702 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:48.689750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:48.689709 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podStartSLOduration=1.689697027 podStartE2EDuration="1.689697027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:48.688385843 +0000 UTC m=+969.413393439" watchObservedRunningTime="2026-11-03 20:13:48.689697027 +0000 UTC m=+969.414704641" Nov 03 20:13:48.807329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:48.807299 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:49.565785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:49.565751 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:49.634978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:49.631235 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:50.451710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:50.451676 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:51.187985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:51.187950 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:51.981068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:51.981034 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:52.426710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.426672 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj"] Nov 03 20:13:52.480139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.469533 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj"] Nov 03 20:13:52.480139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.469718 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.499131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.493666 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:52.499131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.493746 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:13:52.838422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.838329 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrdv\" (UniqueName: \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.838751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.838733 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.838920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.838907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.944248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.944204 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrdv\" (UniqueName: \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.944519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.944502 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.944646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.944623 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.945799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.945772 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.970570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.966078 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:52.996230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:52.979136 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrdv\" (UniqueName: \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\") pod \"maas-api-revoke-sub-f443ba3b-xx2vj\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:53.051835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:53.051803 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:53.983279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:53.977925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:54.778131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:54.738934 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:13:54.912346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:54.907748 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:13:54.973574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:54.972686 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.014130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.007945 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.014130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.012206 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:55.125132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.114395 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj"] Nov 03 20:13:55.628366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.628330 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:55.683177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.683134 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.873062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.873015 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:55.892166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.879352 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.912487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.912440 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.987130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:55.975211 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:13:56.016144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.008761 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:56.016854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.016815 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:56.159132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.122657 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:56.159132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.122775 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:56.159132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.122841 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:56.159132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.122883 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:13:56.159132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.122941 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:13:56.771796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.771758 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" event={"ID":"48363c75-9a78-4f0d-af80-5bf9f9276532","Type":"ContainerStarted","Data":"ad43432b82d76d5ec91d2032f3324ebb5ffffedeb828ef74e28a3aa701ef5596"} Nov 03 20:13:56.914362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.914298 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:56.922622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.896014 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:13:56.996177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:56.983680 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:57.026208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.016513 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:57.063144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.037279 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:13:57.167130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.166394 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:13:57.227794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.227746 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 20:13:57.232638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.232563 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:13:57.232915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.232876 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:13:57.290208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.289654 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z"] Nov 03 20:13:57.299413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.295584 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.309616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.309336 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z"] Nov 03 20:13:57.397547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.397516 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:13:57.407136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.398985 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.407136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.402861 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg8m\" (UniqueName: \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.408949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.407834 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.486940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.486910 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:57.509636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.509606 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.509804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.509744 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.511162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.509780 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg8m\" (UniqueName: \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.511299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.511195 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.524689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.524663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.532659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.532625 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg8m\" (UniqueName: \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\") pod \"maas-api-revoke-sub-7031aae6-pn97z\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.633774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.631525 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:57.652152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.651767 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:57.724302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.724273 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:13:57.784150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.784121 5954 generic.go:356] "Generic (PLEG): container finished" podID="48363c75-9a78-4f0d-af80-5bf9f9276532" containerID="d603ecb7186a160a9ff857c169d8dd2b35e5a6d87c9672493b91c7475c9fd8ac" exitCode=0 Nov 03 20:13:57.785482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.785403 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" event={"ID":"48363c75-9a78-4f0d-af80-5bf9f9276532","Type":"ContainerDied","Data":"d603ecb7186a160a9ff857c169d8dd2b35e5a6d87c9672493b91c7475c9fd8ac"} Nov 03 20:13:57.798144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:57.798040 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:13:58.005591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.005561 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z"] Nov 03 20:13:58.006016 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:13:58.005852 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8eef69_d411_4a94_96db_db7bb1ab4e9a.slice/crio-ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87 WatchSource:0}: Error finding container ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87: Status 404 returned error can't find the container with id ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87 Nov 03 20:13:58.374020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.373989 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:58.793737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.793698 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" event={"ID":"0a8eef69-d411-4a94-96db-db7bb1ab4e9a","Type":"ContainerStarted","Data":"ac63e311aaccae7755e38e40d0f0df645e0a3d0ba87b2e24f493f4d87e9691f4"} Nov 03 20:13:58.793940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.793759 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" event={"ID":"0a8eef69-d411-4a94-96db-db7bb1ab4e9a","Type":"ContainerStarted","Data":"ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87"} Nov 03 20:13:58.863489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.863378 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" podStartSLOduration=2.86335675 podStartE2EDuration="2.86335675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:58.860578738 +0000 UTC m=+979.585586346" watchObservedRunningTime="2026-11-03 20:13:58.86335675 +0000 UTC m=+979.588364347" Nov 03 20:13:58.978578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:58.978546 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:59.140421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.140387 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\" (UniqueName: \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\") pod \"48363c75-9a78-4f0d-af80-5bf9f9276532\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " Nov 03 20:13:59.140863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.140525 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\") pod \"48363c75-9a78-4f0d-af80-5bf9f9276532\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " Nov 03 20:13:59.140863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.140612 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\") pod \"48363c75-9a78-4f0d-af80-5bf9f9276532\" (UID: \"48363c75-9a78-4f0d-af80-5bf9f9276532\") " Nov 03 20:13:59.140957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.140874 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca" pod "48363c75-9a78-4f0d-af80-5bf9f9276532" (UID: "48363c75-9a78-4f0d-af80-5bf9f9276532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:59.141283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.141263 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:59.142660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.142633 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca" pod "48363c75-9a78-4f0d-af80-5bf9f9276532" (UID: "48363c75-9a78-4f0d-af80-5bf9f9276532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:59.142768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.142739 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv" pod "48363c75-9a78-4f0d-af80-5bf9f9276532" (UID: "48363c75-9a78-4f0d-af80-5bf9f9276532"). InnerVolumeSpecName "kube-api-access-xwrdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:59.242681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.242648 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48363c75-9a78-4f0d-af80-5bf9f9276532-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:59.242681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.242675 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrdv\" (UniqueName: \"kubernetes.io/projected/48363c75-9a78-4f0d-af80-5bf9f9276532-kube-api-access-xwrdv\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:59.284788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.284755 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:59.804457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.804380 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" Nov 03 20:13:59.804457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.804391 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj" event={"ID":"48363c75-9a78-4f0d-af80-5bf9f9276532","Type":"ContainerDied","Data":"ad43432b82d76d5ec91d2032f3324ebb5ffffedeb828ef74e28a3aa701ef5596"} Nov 03 20:13:59.804457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.804442 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad43432b82d76d5ec91d2032f3324ebb5ffffedeb828ef74e28a3aa701ef5596" Nov 03 20:13:59.805969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.805945 5954 generic.go:356] "Generic (PLEG): container finished" podID="0a8eef69-d411-4a94-96db-db7bb1ab4e9a" containerID="ac63e311aaccae7755e38e40d0f0df645e0a3d0ba87b2e24f493f4d87e9691f4" exitCode=0 Nov 03 20:13:59.806090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.806019 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" event={"ID":"0a8eef69-d411-4a94-96db-db7bb1ab4e9a","Type":"ContainerDied","Data":"ac63e311aaccae7755e38e40d0f0df645e0a3d0ba87b2e24f493f4d87e9691f4"} Nov 03 20:13:59.985383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:13:59.985352 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:00.259099 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:00.259067 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:14:00.771228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:00.771195 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:01.051745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.051721 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:14:01.058710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.058681 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:14:01.059430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059410 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0a8eef69-d411-4a94-96db-db7bb1ab4e9a" containerName="revoke-keys" Nov 03 20:14:01.059430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059430 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8eef69-d411-4a94-96db-db7bb1ab4e9a" containerName="revoke-keys" Nov 03 20:14:01.059589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059469 5954 cpu_manager.go:414] "containerMap: removing container" podUID="48363c75-9a78-4f0d-af80-5bf9f9276532" containerName="revoke-keys" Nov 03 20:14:01.059589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059477 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48363c75-9a78-4f0d-af80-5bf9f9276532" containerName="revoke-keys" Nov 03 20:14:01.059688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059652 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="48363c75-9a78-4f0d-af80-5bf9f9276532" containerName="revoke-keys" Nov 03 20:14:01.059688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.059679 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8eef69-d411-4a94-96db-db7bb1ab4e9a" containerName="revoke-keys" Nov 03 20:14:01.073389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.063392 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.083456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.083427 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:14:01.105541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.105489 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\") pod \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " Nov 03 20:14:01.105730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.105588 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\") pod \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " Nov 03 20:14:01.105730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.105690 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\" (UniqueName: \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\") pod \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\" (UID: \"0a8eef69-d411-4a94-96db-db7bb1ab4e9a\") " Nov 03 20:14:01.105889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.105862 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca" pod "0a8eef69-d411-4a94-96db-db7bb1ab4e9a" (UID: "0a8eef69-d411-4a94-96db-db7bb1ab4e9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:01.105984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.105957 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106094 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106296 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106410 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106477 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106598 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106640 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106672 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nv6\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106718 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.106869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.106846 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:01.107963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.107935 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m" pod "0a8eef69-d411-4a94-96db-db7bb1ab4e9a" (UID: "0a8eef69-d411-4a94-96db-db7bb1ab4e9a"). InnerVolumeSpecName "kube-api-access-9hg8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:01.108177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.108159 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca" pod "0a8eef69-d411-4a94-96db-db7bb1ab4e9a" (UID: "0a8eef69-d411-4a94-96db-db7bb1ab4e9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:01.208302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208259 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208352 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208449 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208503 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208534 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208596 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208623 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208652 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nv6\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208731 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.208944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208911 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg8m\" (UniqueName: \"kubernetes.io/projected/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-kube-api-access-9hg8m\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:01.208944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208942 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8eef69-d411-4a94-96db-db7bb1ab4e9a-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:01.208944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.208939 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.209284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.209001 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.209284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.209064 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.209284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.209144 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.209592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.209570 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.211003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.210979 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.211240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.211221 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.220885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.220858 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nv6\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.221004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.220983 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.385953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.385924 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:01.564300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.564269 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:14:01.569900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.569878 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.579152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.579122 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:14:01.579508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.579485 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"default-dockercfg-l8wbg\"" Nov 03 20:14:01.579619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.579542 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"kube-root-ca.crt\"" Nov 03 20:14:01.579619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.579562 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"openshift-service-ca.crt\"" Nov 03 20:14:01.579815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.579792 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"distinct-2-w6-d412cf-kserve-self-signed-certs\"" Nov 03 20:14:01.587723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.584814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:01.613945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.613914 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.614084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.614023 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.614100 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.614227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.614214 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.614271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.614261 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.614303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.614283 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s956g\" (UniqueName: \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.730087 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.730577 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.730934 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.731091 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.731220 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.731294 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.731324 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s956g\" (UniqueName: \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.731599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.731430 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.750129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.746676 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:01.763246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.762512 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.763246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.763042 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.806694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.806651 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:01.806927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.806908 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:01.807039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.807025 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:01.814196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.812942 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s956g\" (UniqueName: \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.824611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.824552 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.827718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.826088 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\") pod \"distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:01.844847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.844558 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" event={"ID":"953aca2f-55e6-4c11-b3d7-9a90a842b955","Type":"ContainerStarted","Data":"187e6a36ea6301da80912a0cb9ad6dc55109ac1af9442b944f5f46fe62b92217"} Nov 03 20:14:01.846873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.846839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" event={"ID":"0a8eef69-d411-4a94-96db-db7bb1ab4e9a","Type":"ContainerDied","Data":"ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87"} Nov 03 20:14:01.847096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.847080 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8781318bcd0656db9812d3c3dc4197f335dee975ebd658eed6d5aad2b80b87" Nov 03 20:14:01.847308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.847282 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z" Nov 03 20:14:01.890828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:01.890796 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:02.185627 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:02.185596 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod193e2f79_fd08_47fc_9c1e_ac4829a83dc9.slice/crio-932b73fb39a3d0a319c2379991852a579968051a31d3536fabeb363c621911c0 WatchSource:0}: Error finding container 932b73fb39a3d0a319c2379991852a579968051a31d3536fabeb363c621911c0: Status 404 returned error can't find the container with id 932b73fb39a3d0a319c2379991852a579968051a31d3536fabeb363c621911c0 Nov 03 20:14:02.186571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.186370 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:14:02.463562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.463484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:02.766748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.766666 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp"] Nov 03 20:14:02.771246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.771223 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.779444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.778615 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:02.779444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.779281 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:14:02.781121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.781088 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp"] Nov 03 20:14:02.861595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.861558 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" event={"ID":"953aca2f-55e6-4c11-b3d7-9a90a842b955","Type":"ContainerStarted","Data":"7c19cbfd7c198297885c8761777562728bfe11487f243b7c80c90f580530250e"} Nov 03 20:14:02.863246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.863218 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" event={"ID":"193e2f79-fd08-47fc-9c1e-ac4829a83dc9","Type":"ContainerStarted","Data":"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92"} Nov 03 20:14:02.863363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.863256 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" event={"ID":"193e2f79-fd08-47fc-9c1e-ac4829a83dc9","Type":"ContainerStarted","Data":"932b73fb39a3d0a319c2379991852a579968051a31d3536fabeb363c621911c0"} Nov 03 20:14:02.866183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.866152 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5gt\" (UniqueName: \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.866290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.866277 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.866339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.866305 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.886873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.886831 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podStartSLOduration=1.886815978 podStartE2EDuration="1.886815978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:02.88336382 +0000 UTC m=+983.608371428" watchObservedRunningTime="2026-11-03 20:14:02.886815978 +0000 UTC m=+983.611823579" Nov 03 20:14:02.891801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.891781 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:02.905586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.905541 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podStartSLOduration=1.905528866 podStartE2EDuration="1.905528866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:02.901728468 +0000 UTC m=+983.626736065" watchObservedRunningTime="2026-11-03 20:14:02.905528866 +0000 UTC m=+983.630536462" Nov 03 20:14:02.967813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.967777 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.967813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.967818 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.968303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.968170 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5gt\" (UniqueName: \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.968609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.968581 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.970392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.970367 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:02.979963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:02.979941 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5gt\" (UniqueName: \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\") pod \"maas-api-revoke-sub-e7fa66d1-5d4fp\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:03.095182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.092361 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:03.276057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.275962 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:03.397091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.396702 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:03.485304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.485266 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:03.485886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.485852 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:03.509615 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:03.509589 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eccac9_5a79_41d2_bf90_42fd836b01e6.slice/crio-ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0 WatchSource:0}: Error finding container ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0: Status 404 returned error can't find the container with id ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0 Nov 03 20:14:03.509703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.509633 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp"] Nov 03 20:14:03.872289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:03.872256 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" event={"ID":"b4eccac9-5a79-41d2-bf90-42fd836b01e6","Type":"ContainerStarted","Data":"ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0"} Nov 03 20:14:04.131746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.129212 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:04.386685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.386604 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:04.386685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.386660 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:04.888034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.887998 5954 generic.go:356] "Generic (PLEG): container finished" podID="b4eccac9-5a79-41d2-bf90-42fd836b01e6" containerID="1126f829db9b23dd37b8e049fe8bfdb66a8976530598c1ed13eca87aceb72cda" exitCode=0 Nov 03 20:14:04.888520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.888077 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" event={"ID":"b4eccac9-5a79-41d2-bf90-42fd836b01e6","Type":"ContainerDied","Data":"1126f829db9b23dd37b8e049fe8bfdb66a8976530598c1ed13eca87aceb72cda"} Nov 03 20:14:04.984538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:04.984502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:05.386417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:05.386386 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:05.386581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:05.386438 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:06.076340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.076312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:06.093453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.093431 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:06.211187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.208044 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\") pod \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " Nov 03 20:14:06.211187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.208338 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\") pod \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " Nov 03 20:14:06.211187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.208377 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\" (UniqueName: \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\") pod \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\" (UID: \"b4eccac9-5a79-41d2-bf90-42fd836b01e6\") " Nov 03 20:14:06.243582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.243323 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca" pod "b4eccac9-5a79-41d2-bf90-42fd836b01e6" (UID: "b4eccac9-5a79-41d2-bf90-42fd836b01e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:06.278168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.277589 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca" pod "b4eccac9-5a79-41d2-bf90-42fd836b01e6" (UID: "b4eccac9-5a79-41d2-bf90-42fd836b01e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:06.281272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.280702 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt" pod "b4eccac9-5a79-41d2-bf90-42fd836b01e6" (UID: "b4eccac9-5a79-41d2-bf90-42fd836b01e6"). InnerVolumeSpecName "kube-api-access-dx5gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:06.310353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.310318 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.310353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.310344 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5gt\" (UniqueName: \"kubernetes.io/projected/b4eccac9-5a79-41d2-bf90-42fd836b01e6-kube-api-access-dx5gt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.310513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.310358 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4eccac9-5a79-41d2-bf90-42fd836b01e6-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.387034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.387003 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:06.387222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.387064 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:06.905172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.905138 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" event={"ID":"b4eccac9-5a79-41d2-bf90-42fd836b01e6","Type":"ContainerDied","Data":"ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0"} Nov 03 20:14:06.905325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.905201 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce24ce8b5a24e016097911b4d48259be2cad878b6bddb62d9ed7e0989aee19f0" Nov 03 20:14:06.905325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.905148 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp" Nov 03 20:14:06.906868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:06.906843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:07.387223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:07.387188 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:07.387607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:07.387243 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:07.857417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:07.857388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:08.386789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:08.386756 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:08.386968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:08.386809 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 20:14:08.681005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:08.680900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:09.390518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:09.390485 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:09.391027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:09.391007 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:09.391597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:09.391578 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:14:09.407238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:09.407210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:10.188918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:10.188888 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:10.887092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:10.887058 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:11.604181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:11.604144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:11.891933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:11.891835 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:11.905303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:11.905276 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:12.029746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:12.029718 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:12.346500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:12.346378 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:13.244845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:13.244810 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:14.101812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:14.101734 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:14.416836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:14.416722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:15.097995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:15.097956 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:15.990232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:15.990204 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:16.080181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.080137 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:14:16.081809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.081767 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b4eccac9-5a79-41d2-bf90-42fd836b01e6" containerName="revoke-keys" Nov 03 20:14:16.082015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.081996 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eccac9-5a79-41d2-bf90-42fd836b01e6" containerName="revoke-keys" Nov 03 20:14:16.082336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.082320 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eccac9-5a79-41d2-bf90-42fd836b01e6" containerName="revoke-keys" Nov 03 20:14:16.166735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.166705 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.169931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.169902 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:14:16.305764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.305679 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.305764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.305718 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.305860 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.305943 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.306051 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.306087 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nnx\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.306146 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.306173 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.306218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.306199 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407399 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407468 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407515 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407537 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nnx\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407573 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407608 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407644 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407814 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.407998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.407847 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.408413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.408370 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.408584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.408562 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.408952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.408930 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.409043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.408934 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.409433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.409408 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.410755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.410732 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.411278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.411256 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.420773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.420748 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.420866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.420843 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nnx\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.486530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.486502 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:16.568434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.568353 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:14:16.783783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.783746 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:14:16.783919 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:16.783893 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97a5867_0ca8_4d69_988e_fedf672af29b.slice/crio-c9f0a3ee03d9985d6a974b5b85d3000e207004780473c01d3e8e016b365b083a WatchSource:0}: Error finding container c9f0a3ee03d9985d6a974b5b85d3000e207004780473c01d3e8e016b365b083a: Status 404 returned error can't find the container with id c9f0a3ee03d9985d6a974b5b85d3000e207004780473c01d3e8e016b365b083a Nov 03 20:14:16.786710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.786679 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:16.786791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.786744 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:16.786791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.786775 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:16.809376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:16.809350 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:17.133266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.133226 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" event={"ID":"b97a5867-0ca8-4d69-988e-fedf672af29b","Type":"ContainerStarted","Data":"8e166213a1df294ababda9df37a669b509ab7ad411a9ea38a1d8f9256768c7ef"} Nov 03 20:14:17.133972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.133291 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" event={"ID":"b97a5867-0ca8-4d69-988e-fedf672af29b","Type":"ContainerStarted","Data":"c9f0a3ee03d9985d6a974b5b85d3000e207004780473c01d3e8e016b365b083a"} Nov 03 20:14:17.158328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.158282 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podStartSLOduration=1.1582681830000001 podStartE2EDuration="1.158268183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:17.155062413 +0000 UTC m=+997.880070139" watchObservedRunningTime="2026-11-03 20:14:17.158268183 +0000 UTC m=+997.883275849" Nov 03 20:14:17.498256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.498170 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:17.502451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.502406 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:17.502691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:17.502666 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:18.303986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:18.303942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:18.453560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:18.453214 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 03 20:14:18.501139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:18.500231 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:18.501139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:18.500294 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:19.018132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:19.017837 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:19.418136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:19.417279 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.491769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:19.491728 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:19.492066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:19.492033 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:19.799318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:19.799214 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.001382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.001286 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:20.032331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.032289 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.034119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.034069 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:20.035132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.035099 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:20.047851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.047814 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.288287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.288245 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.437147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.428235 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.437147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.429986 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:14:20.437147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.431232 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.536134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.510416 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:20.553303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.553185 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:20.922217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.904985 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.943768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:20.943671 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:14:21.054250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.054203 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.055758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.055721 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:21.056964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.056938 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:21.058916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.058886 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.100132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.071489 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:21.100132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.071561 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:14:21.100132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.071598 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:14:21.446133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.445504 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.485351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.475318 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:21.520133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.495429 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:14:21.557252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.527834 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:21.557252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.527908 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:21.954292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:21.954233 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.194131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.193506 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:22.194131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.193853 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.236222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.236093 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.253523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.250139 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:22.459576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.459516 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.498754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.498657 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:22.499188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.499055 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:22.527736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.527684 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.566615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.553676 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.716930 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:22.715880 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.195s" Nov 03 20:14:22.913532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.913211 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.975968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:22.975912 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.116676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.116625 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:23.116966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.116934 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.194386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.194282 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:23.196172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.196137 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.238295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.238249 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:23.260821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.260772 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.336184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.327459 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz"] Nov 03 20:14:23.394455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.394419 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.413133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.409559 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:14:23.495582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.472903 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.525629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.506199 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.526098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.526064 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.532022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.526511 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjx7\" (UniqueName: \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.551156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.526661 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:23.584856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.584803 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:23.608583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.530722 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:23.610055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.582452 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz"] Nov 03 20:14:23.610275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.526900 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:23.640130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.634150 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.640130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.634364 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjx7\" (UniqueName: \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.640130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.634502 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.674426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.642247 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.674760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.642615 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:23.674985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.651994 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.719132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.671330 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjx7\" (UniqueName: \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\") pod \"maas-api-revoke-sub-5f11dd46-9dbdz\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:23.935580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:23.935547 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:24.006301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.006248 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.195358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.195247 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:24.220164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.216684 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.252314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.250262 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:24.263399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.263359 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.346131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.343250 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:14:24.346131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.344148 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:14:24.470174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.461839 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:24.470611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.470585 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:14:24.494604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.494572 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:24.494882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.494856 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:24.530515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.530482 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:14:24.551685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.551188 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:14:24.766800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:24.766769 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz"] Nov 03 20:14:24.767484 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:24.767453 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16417dc3_b680_4459_afba_f8abfc7e42fc.slice/crio-079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8 WatchSource:0}: Error finding container 079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8: Status 404 returned error can't find the container with id 079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8 Nov 03 20:14:25.242554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.242529 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:25.460330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.460296 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" event={"ID":"16417dc3-b680-4459-afba-f8abfc7e42fc","Type":"ContainerStarted","Data":"8d54af2508fbdbdb3a5695d0578f6f056f21a43ffa44acabfc039e0e5719e05b"} Nov 03 20:14:25.460497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.460347 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" event={"ID":"16417dc3-b680-4459-afba-f8abfc7e42fc","Type":"ContainerStarted","Data":"079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8"} Nov 03 20:14:25.481987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.481931 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" podStartSLOduration=3.481917869 podStartE2EDuration="3.481917869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:25.477939989 +0000 UTC m=+1006.202947584" watchObservedRunningTime="2026-11-03 20:14:25.481917869 +0000 UTC m=+1006.206925464" Nov 03 20:14:25.487015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.486989 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:25.487156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.487032 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:25.852664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.852622 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:14:25.860163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.860138 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.862208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.862180 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:14:25.868750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.868727 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:14:25.874635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.874555 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.877164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.877143 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:14:25.890075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890049 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890116 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890141 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jqz\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890170 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890210 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890246 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890359 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890430 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890506 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890543 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890584 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890656 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890739 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g567m\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890768 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.890921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890823 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.890968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890950 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.891036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.890990 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.891094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.891074 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992464 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992510 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992536 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992558 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992576 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992614 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.992654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992645 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992687 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992768 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992853 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g567m\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992882 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992893 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.992973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992920 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.992974 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993004 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993054 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993137 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993167 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993186 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jqz\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993270 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.993313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993277 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.993730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993704 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.993797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.994009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.993986 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.994175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.994042 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.994175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.994067 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.994344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.994184 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.995064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.995037 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.995548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.995508 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.995633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.995600 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:25.995825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.995807 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:25.996285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:25.996269 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:26.004443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.004420 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:26.004554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.004502 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:26.004554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.004546 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g567m\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:26.004653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.004638 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jqz\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:26.191059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.190964 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:26.196891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.196852 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:26.197594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.197572 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:26.478074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.478042 5954 generic.go:356] "Generic (PLEG): container finished" podID="16417dc3-b680-4459-afba-f8abfc7e42fc" containerID="8d54af2508fbdbdb3a5695d0578f6f056f21a43ffa44acabfc039e0e5719e05b" exitCode=0 Nov 03 20:14:26.478291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.478271 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" event={"ID":"16417dc3-b680-4459-afba-f8abfc7e42fc","Type":"ContainerDied","Data":"8d54af2508fbdbdb3a5695d0578f6f056f21a43ffa44acabfc039e0e5719e05b"} Nov 03 20:14:26.487371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.487341 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:26.487493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.487391 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:26.705502 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:26.705470 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cbfdf8_dc95_4da3_b374_f413b7186bdb.slice/crio-e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8 WatchSource:0}: Error finding container e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8: Status 404 returned error can't find the container with id e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8 Nov 03 20:14:26.706419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.706183 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:14:26.708715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.708683 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.708814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.708746 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.708814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.708778 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.711348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.711327 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:14:26.712170 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:26.712148 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4458b154_86c1_437c_9e9c_539f67f37f85.slice/crio-ff1e08d44818324451e4487f3ad1386f22060be335c0f08b941f7d9b81cee3e3 WatchSource:0}: Error finding container ff1e08d44818324451e4487f3ad1386f22060be335c0f08b941f7d9b81cee3e3: Status 404 returned error can't find the container with id ff1e08d44818324451e4487f3ad1386f22060be335c0f08b941f7d9b81cee3e3 Nov 03 20:14:26.744744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.744661 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.744911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.744752 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.744911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:26.744794 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:27.194757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.194722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:27.268487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.268454 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6"] Nov 03 20:14:27.326129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.313320 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6"] Nov 03 20:14:27.326129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.313447 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.416615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.416569 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.416784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.416677 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.416784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.416747 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86rq\" (UniqueName: \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.486925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.486833 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:27.487080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.486915 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:27.490271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.490238 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" event={"ID":"23cbfdf8-dc95-4da3-b374-f413b7186bdb","Type":"ContainerStarted","Data":"6dcc59cefe0ddaea4ffcae8f6a4fc112808177c3baa026db90918a266b8f9869"} Nov 03 20:14:27.490435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.490301 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" event={"ID":"23cbfdf8-dc95-4da3-b374-f413b7186bdb","Type":"ContainerStarted","Data":"e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8"} Nov 03 20:14:27.492591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.492558 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" event={"ID":"4458b154-86c1-437c-9e9c-539f67f37f85","Type":"ContainerStarted","Data":"17416d6823128370bb5df90ac6a170068364af759281ce2513f5cd32844f3834"} Nov 03 20:14:27.492712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.492644 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" event={"ID":"4458b154-86c1-437c-9e9c-539f67f37f85","Type":"ContainerStarted","Data":"ff1e08d44818324451e4487f3ad1386f22060be335c0f08b941f7d9b81cee3e3"} Nov 03 20:14:27.517012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.516940 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podStartSLOduration=2.5169193549999997 podStartE2EDuration="2.516919355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.512582667 +0000 UTC m=+1008.237590265" watchObservedRunningTime="2026-11-03 20:14:27.516919355 +0000 UTC m=+1008.241926950" Nov 03 20:14:27.518663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.518630 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.518801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.518712 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.518801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.518788 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86rq\" (UniqueName: \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.525438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.525410 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.526305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.526281 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.550524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.550466 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podStartSLOduration=2.550449229 podStartE2EDuration="2.550449229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.550400822 +0000 UTC m=+1008.275408420" watchObservedRunningTime="2026-11-03 20:14:27.550449229 +0000 UTC m=+1008.275456826" Nov 03 20:14:27.552272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.552245 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86rq\" (UniqueName: \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\") pod \"maas-api-revoke-sub-0390c9ab-xmrz6\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.741242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.733407 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:27.886283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.885997 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:27.992671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.992578 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\" (UniqueName: \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\") pod \"16417dc3-b680-4459-afba-f8abfc7e42fc\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " Nov 03 20:14:27.992903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.992883 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\") pod \"16417dc3-b680-4459-afba-f8abfc7e42fc\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " Nov 03 20:14:27.993329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:27.993310 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\") pod \"16417dc3-b680-4459-afba-f8abfc7e42fc\" (UID: \"16417dc3-b680-4459-afba-f8abfc7e42fc\") " Nov 03 20:14:28.007155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.002281 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca" pod "16417dc3-b680-4459-afba-f8abfc7e42fc" (UID: "16417dc3-b680-4459-afba-f8abfc7e42fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:28.091485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.091448 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca" pod "16417dc3-b680-4459-afba-f8abfc7e42fc" (UID: "16417dc3-b680-4459-afba-f8abfc7e42fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:28.101191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.095324 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7" pod "16417dc3-b680-4459-afba-f8abfc7e42fc" (UID: "16417dc3-b680-4459-afba-f8abfc7e42fc"). InnerVolumeSpecName "kube-api-access-2mjx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:28.101191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.100841 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:28.101191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.100869 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjx7\" (UniqueName: \"kubernetes.io/projected/16417dc3-b680-4459-afba-f8abfc7e42fc-kube-api-access-2mjx7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:28.101191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.100886 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16417dc3-b680-4459-afba-f8abfc7e42fc-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:28.180493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.180258 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:28.192088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.192015 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:28.192958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.192930 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.193073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.192999 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:28.278310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.254533 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:28.473366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.473319 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.473710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.473600 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:28.492643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.492612 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.492840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.492810 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:14:28.504420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.504326 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6"] Nov 03 20:14:28.568981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.567210 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" Nov 03 20:14:28.568981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.568319 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz" event={"ID":"16417dc3-b680-4459-afba-f8abfc7e42fc","Type":"ContainerDied","Data":"079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8"} Nov 03 20:14:28.568981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:28.568450 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="079ee653ce6b89e724139205bb353d72283dfedcc769f8fe7e42acb37b4fa6c8" Nov 03 20:14:29.053127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.052427 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:29.197066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.197032 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.197605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.197476 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:29.229000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.213247 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.229000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.213319 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:29.593302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.593216 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:29.595808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.595779 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:29.597247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.597222 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:14:29.601137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.601096 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" event={"ID":"151fee40-ffd7-4c60-a990-5de55fc39cad","Type":"ContainerStarted","Data":"08a1545ae88690fa82b11a0ebd3daf363ded5df40fd4ab2dbb5403c894251a53"} Nov 03 20:14:29.601228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.601155 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" event={"ID":"151fee40-ffd7-4c60-a990-5de55fc39cad","Type":"ContainerStarted","Data":"00267ce62f6883e4f6bcd98569868819abeb8fdbfdc6c0217d6864429ea56a3a"} Nov 03 20:14:29.695173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.695075 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" podStartSLOduration=2.69505623 podStartE2EDuration="2.69505623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:29.690556268 +0000 UTC m=+1010.415563866" watchObservedRunningTime="2026-11-03 20:14:29.69505623 +0000 UTC m=+1010.420063827" Nov 03 20:14:29.848197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:29.845006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:30.191614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.191574 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.191771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.191632 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:30.198513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.198486 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.198662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.198536 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:30.290724 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:30.290690 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:14:30.598603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.598520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:30.612497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.612465 5954 generic.go:356] "Generic (PLEG): container finished" podID="151fee40-ffd7-4c60-a990-5de55fc39cad" containerID="08a1545ae88690fa82b11a0ebd3daf363ded5df40fd4ab2dbb5403c894251a53" exitCode=0 Nov 03 20:14:30.612666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:30.612653 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" event={"ID":"151fee40-ffd7-4c60-a990-5de55fc39cad","Type":"ContainerDied","Data":"08a1545ae88690fa82b11a0ebd3daf363ded5df40fd4ab2dbb5403c894251a53"} Nov 03 20:14:31.124568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.124533 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:14:31.137129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.125922 5954 cpu_manager.go:414] "containerMap: removing container" podUID="16417dc3-b680-4459-afba-f8abfc7e42fc" containerName="revoke-keys" Nov 03 20:14:31.137372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.137349 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16417dc3-b680-4459-afba-f8abfc7e42fc" containerName="revoke-keys" Nov 03 20:14:31.137764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.137744 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="16417dc3-b680-4459-afba-f8abfc7e42fc" containerName="revoke-keys" Nov 03 20:14:31.145383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.145360 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.187536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.187502 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:14:31.191459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.191428 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.191588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.191486 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:31.195086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.195039 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:14:31.198174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.198149 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.198283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.198192 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:31.208489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.208311 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.217128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.214366 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:14:31.296761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.296729 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:14:31.301402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.301372 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.349862 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361274 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361356 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361418 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361451 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361515 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhcq\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361541 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361578 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361719 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361779 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361809 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361841 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361871 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361914 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361945 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.361977 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.362032 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.362072 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.365132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.362123 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgpt\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.412351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.412270 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.412933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.412276 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:14:31.473541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473499 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.473719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473558 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.473719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473607 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.473719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473646 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.473719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473679 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgpt\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.473923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473729 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.473923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473776 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftqs\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.473923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473840 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.473923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473881 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.474131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.473935 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.474131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474002 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.474131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474037 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.474131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474060 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.474335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474145 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhcq\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.474335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474171 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.474582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.474556 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.475535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.475506 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.475759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.475622 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.475759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.475752 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.476477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.476449 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.477618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.477593 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.477943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.477911 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479063 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.479239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.478789 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.479315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.478915 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.478418 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479747 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.479845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479782 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479803 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.479845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479828 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479851 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.479941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479866 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.479941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479886 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.479941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479919 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.479941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.479935 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.480353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.480316 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.480729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.480710 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.481656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.481584 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.482679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.482663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.489496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.489471 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:14:31.489626 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.489538 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:14:31.495711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.495687 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.496899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.496663 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.496899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.496863 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.497384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.497170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.497888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.497866 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgpt\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.499928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.499603 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhcq\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.501448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.501427 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:14:31.504668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.504646 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.581186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581154 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.581384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581209 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.581384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581229 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.581384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581245 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.581384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581271 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftqs\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.581384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581298 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.581659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581633 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.581713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581668 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.581772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581755 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.581823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581812 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.581877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581834 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.581929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581886 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmkgq\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.581929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581933 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.582091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.581977 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.582091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582011 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582031 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582146 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582247 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582296 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582327 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582333 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.582538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582403 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582439 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582467 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582522 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582552 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582585 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvkv\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.582723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582621 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.582723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582625 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.582933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.582914 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.583290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.583273 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.583632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.583605 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.585255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.585234 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.588185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.588166 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:31.594050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.594023 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftqs\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.594357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.594337 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.683837 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.683881 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.683951 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684020 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684141 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684176 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684238 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmkgq\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684271 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684296 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684362 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684397 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684454 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684608 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684676 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684737 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvkv\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684782 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.684842 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685043 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685077 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:31.685312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685125 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685570 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685754 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.685882 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.686163 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.686474 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.686540 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.686823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.686769 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.687137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.686875 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.688164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.687853 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.688713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.688690 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.690164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.690144 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.690434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.690379 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.697885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.697859 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvkv\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.700514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.700466 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.701519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.701497 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.707499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.706833 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmkgq\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:31.797628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.797310 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:31.923196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.899323 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:31.923196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.899461 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:31.923196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:31.900267 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:32.174814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.174704 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:14:32.193246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.192262 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.193246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.192319 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:32.199544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.197876 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.199544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.197919 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:32.231169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.223301 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.231169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.223398 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.231169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.223443 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.498962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.497304 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:14:32.498962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.497863 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.498962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.497948 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.498962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.497991 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.501653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.501331 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:32.608828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.608798 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\") pod \"151fee40-ffd7-4c60-a990-5de55fc39cad\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " Nov 03 20:14:32.608934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.608857 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\") pod \"151fee40-ffd7-4c60-a990-5de55fc39cad\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " Nov 03 20:14:32.609281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.609262 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\" (UniqueName: \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\") pod \"151fee40-ffd7-4c60-a990-5de55fc39cad\" (UID: \"151fee40-ffd7-4c60-a990-5de55fc39cad\") " Nov 03 20:14:32.609669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.609640 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca" pod "151fee40-ffd7-4c60-a990-5de55fc39cad" (UID: "151fee40-ffd7-4c60-a990-5de55fc39cad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:32.609985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.609966 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:32.619331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.619274 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq" pod "151fee40-ffd7-4c60-a990-5de55fc39cad" (UID: "151fee40-ffd7-4c60-a990-5de55fc39cad"). InnerVolumeSpecName "kube-api-access-j86rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:32.676266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.676217 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca" pod "151fee40-ffd7-4c60-a990-5de55fc39cad" (UID: "151fee40-ffd7-4c60-a990-5de55fc39cad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:32.686632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.686604 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" event={"ID":"151fee40-ffd7-4c60-a990-5de55fc39cad","Type":"ContainerDied","Data":"00267ce62f6883e4f6bcd98569868819abeb8fdbfdc6c0217d6864429ea56a3a"} Nov 03 20:14:32.687030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.686662 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00267ce62f6883e4f6bcd98569868819abeb8fdbfdc6c0217d6864429ea56a3a" Nov 03 20:14:32.687030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.686741 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6" Nov 03 20:14:32.690514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.690486 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" event={"ID":"bd479ced-6534-4776-a30f-c0e56231a0a8","Type":"ContainerStarted","Data":"62b2dc22e8306c1ffcd1435f64cb0bd3eab74cfd5bb1adfa5420b51cde3e771c"} Nov 03 20:14:32.692866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.692139 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" event={"ID":"74ea9a19-fb71-4628-804a-092f6cea7f7e","Type":"ContainerStarted","Data":"e87f3d3782cbe705fc9c3f2ac4f2c6e7663db05ecabfac63733c5dcaa27f1dad"} Nov 03 20:14:32.719140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.714779 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151fee40-ffd7-4c60-a990-5de55fc39cad-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:32.719140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.714815 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86rq\" (UniqueName: \"kubernetes.io/projected/151fee40-ffd7-4c60-a990-5de55fc39cad-kube-api-access-j86rq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:32.813267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.813188 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:14:32.876042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.869687 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.876042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.869782 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.876042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.869828 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.909825 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:32.909781 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fb06eb_ee08_4fd4_8bd9_8eca9fdd4d2b.slice/crio-bb9055a0a40280338d90f3458d8c9a5d61bdf195548159d513848faf8d8ed11c WatchSource:0}: Error finding container bb9055a0a40280338d90f3458d8c9a5d61bdf195548159d513848faf8d8ed11c: Status 404 returned error can't find the container with id bb9055a0a40280338d90f3458d8c9a5d61bdf195548159d513848faf8d8ed11c Nov 03 20:14:32.911908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.911626 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:14:32.983247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:32.983213 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:14:33.029657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.009650 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.029657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.009756 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.029657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.009805 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.332327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.288063 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.357129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.334886 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:33.357129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.347635 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.357129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.347783 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.357129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.347865 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:33.399134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.380362 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.399134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.380431 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:33.409132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.403292 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:33.761883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.760479 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" event={"ID":"f8acdae3-b9d5-4a0d-890c-47e5705643ae","Type":"ContainerStarted","Data":"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c"} Nov 03 20:14:33.761883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.760556 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" event={"ID":"f8acdae3-b9d5-4a0d-890c-47e5705643ae","Type":"ContainerStarted","Data":"1c2e362545d7ab7895c17f6f1b98376b3b9695893116628caa8925aee8f7edc4"} Nov 03 20:14:33.823154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.821903 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" event={"ID":"74ea9a19-fb71-4628-804a-092f6cea7f7e","Type":"ContainerStarted","Data":"41bb2dff1c83efcb984bdc79c1c5070f07c49160f4b66b285d29d9e7a07ed129"} Nov 03 20:14:33.834242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.834205 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" event={"ID":"bd479ced-6534-4776-a30f-c0e56231a0a8","Type":"ContainerStarted","Data":"abce18e69927634c2e3f0614986ba46e0ef3af29e7d9814676dac45706619240"} Nov 03 20:14:33.849373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.849246 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podStartSLOduration=2.849225998 podStartE2EDuration="2.849225998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.823370119 +0000 UTC m=+1014.548377717" watchObservedRunningTime="2026-11-03 20:14:33.849225998 +0000 UTC m=+1014.574233594" Nov 03 20:14:33.866297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.866240 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" event={"ID":"2b57a418-15d9-488d-9007-9647debd6e6f","Type":"ContainerStarted","Data":"8d8d0ce0e91b9ceb6b434386e179cbee06db31c16e3115c28e46eb2c2bf4983b"} Nov 03 20:14:33.866695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.866625 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" event={"ID":"2b57a418-15d9-488d-9007-9647debd6e6f","Type":"ContainerStarted","Data":"d6fb5befd2e6f0aeed73ce0d7e268e8b0d5e6f048cb543a38253f55cc639bcfa"} Nov 03 20:14:33.871273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.871243 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" event={"ID":"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b","Type":"ContainerStarted","Data":"bb9055a0a40280338d90f3458d8c9a5d61bdf195548159d513848faf8d8ed11c"} Nov 03 20:14:33.874412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.874361 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podStartSLOduration=2.874343801 podStartE2EDuration="2.874343801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.872096925 +0000 UTC m=+1014.597104521" watchObservedRunningTime="2026-11-03 20:14:33.874343801 +0000 UTC m=+1014.599351401" Nov 03 20:14:33.944756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.944672 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:33.984129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:33.979156 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podStartSLOduration=2.979133052 podStartE2EDuration="2.979133052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.975955792 +0000 UTC m=+1014.700963419" watchObservedRunningTime="2026-11-03 20:14:33.979133052 +0000 UTC m=+1014.704140646" Nov 03 20:14:34.063144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.061474 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podStartSLOduration=3.06144814 podStartE2EDuration="3.06144814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:33.943305822 +0000 UTC m=+1014.668313419" watchObservedRunningTime="2026-11-03 20:14:34.06144814 +0000 UTC m=+1014.786455736" Nov 03 20:14:34.220726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.220689 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.221006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.220978 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:14:34.221611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.221585 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.221782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.221758 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 20:14:34.624131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.601438 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:34.624131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.603770 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.624131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.603829 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:34.701130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.701015 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:34.701331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.701221 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.701331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.701281 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:34.805130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.804465 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:34.805130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.804606 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.805130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.804666 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:34.911212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.910417 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.911212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.910487 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:34.940132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.939962 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" event={"ID":"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b","Type":"ContainerStarted","Data":"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4"} Nov 03 20:14:34.971402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:34.971355 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.028846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.028780 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podStartSLOduration=4.028760549 podStartE2EDuration="4.028760549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:34.995987809 +0000 UTC m=+1015.720995409" watchObservedRunningTime="2026-11-03 20:14:35.028760549 +0000 UTC m=+1015.753768146" Nov 03 20:14:35.067290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.067245 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.199580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.199476 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.262517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.262472 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.331878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.331840 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:35.637267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.637204 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:35.705131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.645290 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.705131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.697591 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:35.741835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.741794 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.742143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.742094 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:35.841132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.835336 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.841132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.835398 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:35.911220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.901637 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.911476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.911438 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:35.957131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.928678 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:35.961233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.937769 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.978327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.978269 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:35.978678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:35.978653 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.000676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.000624 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:14:36.114132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.096483 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.155346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.155305 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:36.235327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.228502 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:36.269163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.237537 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:36.269163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.244209 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:36.269163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.244275 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.302268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.295866 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:36.302268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.297181 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:36.302268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.297225 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.702846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.702734 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:36.722186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.704383 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.722186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.704435 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:36.722186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.704653 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.722186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:36.705801 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:14:37.059354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.059244 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:37.085735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.085703 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.085969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.085942 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:37.086180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.086162 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.086331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.086311 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:37.086640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.086612 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.086772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.086753 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:37.087854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.087834 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:37.103131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.091054 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.103131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.091189 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:37.105952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.105910 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:37.108167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.108146 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:37.108433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.108416 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:37.119667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.119643 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:37.142129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.142084 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:14:37.142564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.142543 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:14:37.248273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.248211 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 20:14:37.248548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.248516 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:37.306589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.306544 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:37.307461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.307365 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:37.307724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.307705 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:14:37.694495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.694453 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.694685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.694520 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:37.699144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.696839 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.699144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.696886 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:37.826352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:37.826263 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:38.126154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.048164 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.143946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.067404 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:38.396043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.107416 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:38.399401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.399368 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:38.406655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.271934 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.406907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.406856 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:38.521615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.396398 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.521944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.521896 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:38.522150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.393715 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.526371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.266739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:38.721189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.658402 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 03 20:14:38.721189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.666231 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.721189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.666295 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:38.737282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.737229 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.996188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.979285 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.996188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:38.979350 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:39.008129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.008065 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.008391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.008352 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:39.008922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.008896 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.009136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.009095 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:39.136995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.116344 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.136995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.116408 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:39.147163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.145003 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.181251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.147866 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.232559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.232519 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.232815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.232780 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:39.238559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.238517 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:39.396402 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:39.325642 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.927s" Nov 03 20:14:39.396402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.393336 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:14:39.396402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.393380 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:14:39.423174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.406247 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.423174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.419281 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:39.436002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.435901 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:39.568572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.567290 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.916337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.831630 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.916337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.831695 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:39.946390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:39.946343 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:40.077131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.047567 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.077526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.077474 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:40.080081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.080041 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.080269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.080245 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:40.130134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.115635 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.130134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.115701 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:40.172648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.151542 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.172648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.151612 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:40.237127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.226190 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.237127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.227657 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:40.237127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.227754 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:40.406096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.406049 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.406410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.406381 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.610680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.600905 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.610680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.600957 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:40.659246 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:40.659213 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.24s" Nov 03 20:14:40.659530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.659512 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:40.708051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.708020 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.719956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.719919 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:40.845681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.801986 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.845681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.802047 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:40.845681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.803068 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:40.893285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.892950 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:40.925140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.920363 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.925140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.920421 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:40.925140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.920494 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.925140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:40.920515 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:41.160210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.151736 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.380132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.372287 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.380132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.373054 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:41.414221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.407674 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.613135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.596069 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.613135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.596145 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:41.743511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.743325 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.771130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.743698 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:41.815188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.811588 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.838744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.838692 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:41.856829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.852833 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.856829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.852899 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:41.970132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.919213 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.991292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.991248 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:42.001185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.924280 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.021496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:41.923386 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.112309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.109490 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:42.267187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.257339 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:42.299250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.299200 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podUID="3602a4e5-7b74-44e0-a640-3e608c97bf19" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.400183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.400140 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:42.431135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.403027 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.439422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.431468 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.439679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.431502 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:42.457097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.431550 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:42.591721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.591628 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.768647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.703650 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.769453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.768908 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:42.770009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.768588 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:42.818384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.818349 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.818640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.818614 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:42.919151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.918086 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:42.962136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.929469 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.962136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.929525 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:42.962136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.951089 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.962136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.960280 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.962136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:42.960346 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:43.145131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.122208 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.313920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.276096 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.433404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.433363 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:43.453592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.452246 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.482772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.454927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:43.487268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.462329 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:43.504135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.503219 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.737841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.737801 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.738146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.738097 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:43.925561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.922286 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.926127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.925819 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.926127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.925870 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:43.926127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.925941 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.926127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:43.925963 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:44.130291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.113240 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.130291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.113301 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:44.177291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.177255 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.177569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.177535 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:44.185734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.185701 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:44.371251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.371204 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:44.451826 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:14:44.451720 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.015s" Nov 03 20:14:44.470259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.470221 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:44.474675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.474649 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:14:44.477099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.477064 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:44.481388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.481174 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:44.489227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.489179 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:44.509233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.509192 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:44.777209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.756458 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.801651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.801613 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 20:14:44.801923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.774538 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:14:44.827187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.764196 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.827187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.805945 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:44.827187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.771168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:44.866195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.866148 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.936027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.925488 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:14:44.936752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.928650 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.936959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.936935 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:44.937130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.928746 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.937266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.937247 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 20:14:44.937445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:44.935887 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:45.048115 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.048007 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:14:45.236129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.231421 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:14:45.377438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.377396 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:45.434880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.434684 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:45.478442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.478402 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:45.555962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.555927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:45.589529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.589488 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 20:14:45.589832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.589797 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 20:14:45.720642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.720542 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:45.732985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.732957 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:45.736835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.736806 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:14:45.817913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.817877 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:45.841520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.841476 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:14:45.863386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.863356 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht"] Nov 03 20:14:45.864657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.864621 5954 cpu_manager.go:414] "containerMap: removing container" podUID="151fee40-ffd7-4c60-a990-5de55fc39cad" containerName="revoke-keys" Nov 03 20:14:45.864806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.864793 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151fee40-ffd7-4c60-a990-5de55fc39cad" containerName="revoke-keys" Nov 03 20:14:45.865100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.865086 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="151fee40-ffd7-4c60-a990-5de55fc39cad" containerName="revoke-keys" Nov 03 20:14:45.891158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.881815 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:45.893295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.888663 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht"] Nov 03 20:14:45.899463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.899433 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:45.906128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.900208 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:14:45.930168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.928376 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:14:45.930168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.928438 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 03 20:14:45.945918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.942642 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:45.946688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.946655 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:45.946921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:45.946901 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmw76\" (UniqueName: \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.049219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.049095 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.049526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.049486 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmw76\" (UniqueName: \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.049711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.049684 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.050832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.050792 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.076375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.076341 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.076877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.076856 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmw76\" (UniqueName: \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\") pod \"maas-api-revoke-sub-34d0897a-bwqht\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.267513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.267476 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:46.324931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.324901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:46.595022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.594911 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:46.800038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.799223 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:46.875272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.873872 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:46.876685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.875548 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:46.878091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.878072 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:14:46.907616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.907245 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:46.925486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.920360 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:46.980646 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:46.980600 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99c54ac_01a4_4b16_b2e4_d3f6f5421b92.slice/crio-21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6 WatchSource:0}: Error finding container 21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6: Status 404 returned error can't find the container with id 21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6 Nov 03 20:14:46.986809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:46.986786 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht"] Nov 03 20:14:47.169817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.169742 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:47.264661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.264620 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:47.358203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.345484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:47.392162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.391692 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" event={"ID":"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92","Type":"ContainerStarted","Data":"21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6"} Nov 03 20:14:47.464236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.464145 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:14:47.464406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.464242 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:14:47.464406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.464294 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:14:47.474919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:47.474807 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:48.290927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:48.290890 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:48.417343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:48.415138 5954 generic.go:356] "Generic (PLEG): container finished" podID="e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" containerID="5361cf7d72fed7bb83566bbd8ef344c2c2a07231089756c9863fad3a01e7d02d" exitCode=0 Nov 03 20:14:48.417343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:48.416748 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" event={"ID":"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92","Type":"ContainerDied","Data":"5361cf7d72fed7bb83566bbd8ef344c2c2a07231089756c9863fad3a01e7d02d"} Nov 03 20:14:49.097046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.097007 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:49.607943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.607917 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:49.636358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.636324 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\") pod \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " Nov 03 20:14:49.636578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.636557 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\" (UniqueName: \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\") pod \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " Nov 03 20:14:49.636768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.636747 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\") pod \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\" (UID: \"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92\") " Nov 03 20:14:49.637272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.637248 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca" pod "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" (UID: "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:49.637746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.637718 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:49.639552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.639520 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca" pod "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" (UID: "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:49.640557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.640530 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76" pod "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" (UID: "e99c54ac-01a4-4b16-b2e4-d3f6f5421b92"). InnerVolumeSpecName "kube-api-access-bmw76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:49.740672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.740581 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:49.740672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:49.740611 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmw76\" (UniqueName: \"kubernetes.io/projected/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92-kube-api-access-bmw76\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:50.095207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.088118 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:50.101130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.099175 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82"] Nov 03 20:14:50.101130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.100435 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" containerName="revoke-keys" Nov 03 20:14:50.101130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.100460 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" containerName="revoke-keys" Nov 03 20:14:50.101130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.100780 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" containerName="revoke-keys" Nov 03 20:14:50.123127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.121878 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.144130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.138774 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82"] Nov 03 20:14:50.171162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.156729 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cd8l\" (UniqueName: \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.171810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.171776 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.184142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.183818 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.286252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.286216 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cd8l\" (UniqueName: \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.286418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.286394 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.286418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.286414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.287198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.287176 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.288793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.288774 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.312994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.312968 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cd8l\" (UniqueName: \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\") pod \"maas-api-revoke-sub-439024a5-q2k82\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.433325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.433290 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" event={"ID":"e99c54ac-01a4-4b16-b2e4-d3f6f5421b92","Type":"ContainerDied","Data":"21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6"} Nov 03 20:14:50.433517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.433324 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht" Nov 03 20:14:50.433517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.433338 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b509186eb9c10a36b759e4fa75d837e9591e805a47c4fa8742da9fd54a1aa6" Nov 03 20:14:50.498324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.498292 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:50.791351 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:14:50.791309 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc91c4d2_914a_477d_a9ab_1f4039fb6b4c.slice/crio-a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc WatchSource:0}: Error finding container a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc: Status 404 returned error can't find the container with id a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc Nov 03 20:14:50.792010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:50.791965 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82"] Nov 03 20:14:51.012350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:51.012318 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:51.444121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:51.444080 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" event={"ID":"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c","Type":"ContainerStarted","Data":"d40f45d5aa5bc543a0d706d3baf0690a5aeb2a6f0cf7c38744be68ff5433d774"} Nov 03 20:14:51.444295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:51.444156 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" event={"ID":"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c","Type":"ContainerStarted","Data":"a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc"} Nov 03 20:14:51.470860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:51.470815 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" podStartSLOduration=1.4708009340000001 podStartE2EDuration="1.470800934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:51.467008142 +0000 UTC m=+1032.192015739" watchObservedRunningTime="2026-11-03 20:14:51.470800934 +0000 UTC m=+1032.195808529" Nov 03 20:14:51.990409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:51.990332 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:52.455775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:52.455734 5954 generic.go:356] "Generic (PLEG): container finished" podID="bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" containerID="d40f45d5aa5bc543a0d706d3baf0690a5aeb2a6f0cf7c38744be68ff5433d774" exitCode=0 Nov 03 20:14:52.455982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:52.455793 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" event={"ID":"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c","Type":"ContainerDied","Data":"d40f45d5aa5bc543a0d706d3baf0690a5aeb2a6f0cf7c38744be68ff5433d774"} Nov 03 20:14:54.113138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.110240 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.206132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.198012 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.385698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.385570 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.492715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.469480 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:54.492715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.478188 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:54.713193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.621708 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:54.718470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.714714 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.890132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:54.856309 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.247246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.247196 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:55.249824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.249792 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:55.258138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.251482 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:14:55.327242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.324026 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:14:55.418191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.409465 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.418191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.411359 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:14:55.470077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.470035 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.480224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.479128 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.494391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.494361 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:14:55.537516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.537441 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:14:55.554133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.531350 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:55.610251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.537350 5954 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:55.610505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.610475 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.697597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.697555 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\") pod \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " Nov 03 20:14:55.753706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.753663 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.804193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.793632 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:14:55.880130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.838708 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\") pod \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " Nov 03 20:14:55.902414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.902361 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\" (UniqueName: \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\") pod \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\" (UID: \"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c\") " Nov 03 20:14:55.916423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.916379 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca" pod "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" (UID: "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:55.972320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.961421 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l" pod "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" (UID: "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c"). InnerVolumeSpecName "kube-api-access-8cd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:55.987269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.987235 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:55.987517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:55.987495 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cd8l\" (UniqueName: \"kubernetes.io/projected/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-kube-api-access-8cd8l\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:56.012490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.005433 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.030171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.013632 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:14:56.051153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.033683 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca" pod "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" (UID: "bc91c4d2-914a-477d-a9ab-1f4039fb6b4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:56.200133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.185471 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:56.261928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.261892 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:56.263600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.262591 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:56.338017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.337974 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" event={"ID":"bc91c4d2-914a-477d-a9ab-1f4039fb6b4c","Type":"ContainerDied","Data":"a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc"} Nov 03 20:14:56.544310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.544219 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3cd6d38ab9e251e78ae82931ea4cf910d17ae2fdd7cc812a5daeb355070c4dc" Nov 03 20:14:56.544642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.412219 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:56.592158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.494950 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:56.605172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.496240 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:56.605172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.359817 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82" Nov 03 20:14:56.796306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:56.796200 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:57.006246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.006202 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:57.335316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.328505 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:57.335316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.330933 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 20:14:57.335316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.331044 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:57.652133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.637458 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:57.652133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.637826 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:57.652133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.638144 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:57.829297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:57.829253 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:58.011692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.011567 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.409882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.409830 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:58.435264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.435215 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:14:58.562281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.562234 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.644131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.641629 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.677191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.644740 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.704130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.645313 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:58.704303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.645411 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.875083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.875040 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.214131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:58.918333 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 03 20:14:59.214131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.159442 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:14:59.433654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.414231 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:14:59.440401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.440365 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.673132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.665647 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:14:59.714132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.713468 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:59.714132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:14:59.713825 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:00.214279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.214235 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.214820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.214798 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:00.216140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.216119 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" containerName="revoke-keys" Nov 03 20:15:00.216278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.216265 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" containerName="revoke-keys" Nov 03 20:15:00.216719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.216701 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" containerName="revoke-keys" Nov 03 20:15:00.465709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.231229 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:00.466557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.293624 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:00.474139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.474091 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.475264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.475240 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:15:00.475374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.475349 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:15:00.475374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.475369 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:00.475688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.392417 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:00.480622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.477209 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.481475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.481460 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:15:00.481616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.481606 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:15:00.481710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.481702 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:15:00.481807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.481799 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:15:00.482550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.482533 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.487870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.484554 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:00.492171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.491556 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:00.495569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.495216 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:15:00.498182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.496202 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:00.498576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.498543 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.516256 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:00.516222 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:15:00.516880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.516855 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:00.516934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.516908 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:00.517719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.517701 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.523805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.523771 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:00.523906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.523877 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.527605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.527580 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:00.532133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.532100 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:00.539643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.539619 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:00.611348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.611307 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2dt\" (UniqueName: \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.611536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.611420 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.611930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.611904 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.612098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612075 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.612204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612192 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.612278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612246 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd7m\" (UniqueName: \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.612338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612319 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65rg\" (UniqueName: \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.612432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612413 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.612844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612813 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgtp\" (UniqueName: \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.612929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.612876 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9lc\" (UniqueName: \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.714337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714305 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2dt\" (UniqueName: \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.714534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714358 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.714534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714454 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.714534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714478 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.714534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714529 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.714769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714570 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd7m\" (UniqueName: \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.714886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714859 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65rg\" (UniqueName: \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.714997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.714978 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.715240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715219 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.715334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715302 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.715377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715313 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.715590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715575 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.715655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715642 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgtp\" (UniqueName: \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.715707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715676 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9lc\" (UniqueName: \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.715837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.715792 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.733024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.732992 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65rg\" (UniqueName: \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:00.733189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.733033 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2dt\" (UniqueName: \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.736764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.736696 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgtp\" (UniqueName: \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\") pod \"maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.740770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.740742 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd7m\" (UniqueName: \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\") pod \"maas-api-key-cleanup-29895615-gs722\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.741534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.741505 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9lc\" (UniqueName: \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\") pod \"maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.823484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.823451 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:00.833631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.833605 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:00.842824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.842793 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:00.852926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.852895 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:00.861874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:00.861706 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:01.354670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.354645 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:01.355735 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:01.355693 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9638c8b_0de1_4163_b750_7dd3edf6d607.slice/crio-1accb08d01691d0f1bfabc2508a443dc655eadc9f3edd96364f1864010507522 WatchSource:0}: Error finding container 1accb08d01691d0f1bfabc2508a443dc655eadc9f3edd96364f1864010507522: Status 404 returned error can't find the container with id 1accb08d01691d0f1bfabc2508a443dc655eadc9f3edd96364f1864010507522 Nov 03 20:15:01.356768 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:01.356740 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440fac0d_03d3_41b9_bf3b_823b357e3c09.slice/crio-b62549525eb65e90f43cb50ab2f98c3d2c963a290a27fffe19b8d854f770897e WatchSource:0}: Error finding container b62549525eb65e90f43cb50ab2f98c3d2c963a290a27fffe19b8d854f770897e: Status 404 returned error can't find the container with id b62549525eb65e90f43cb50ab2f98c3d2c963a290a27fffe19b8d854f770897e Nov 03 20:15:01.360575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.360495 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:01.383792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.383768 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:01.419325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.419297 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:01.450039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.450006 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerStarted","Data":"2cf872c7b7872051eb35232cd3206fa8e4c35f1322435797f45fc3b0e55d357f"} Nov 03 20:15:01.451634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.451610 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerStarted","Data":"b62549525eb65e90f43cb50ab2f98c3d2c963a290a27fffe19b8d854f770897e"} Nov 03 20:15:01.453179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.453152 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerStarted","Data":"37b92719483fe49261d465ba301f5a2a01370faa397dd2c9a0a93dbba53d6f76"} Nov 03 20:15:01.454493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.454474 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerStarted","Data":"1accb08d01691d0f1bfabc2508a443dc655eadc9f3edd96364f1864010507522"} Nov 03 20:15:01.455150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.455132 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:01.518602 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:01.518564 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436cd249_ff1f_4b17_b6b2_3592fba16020.slice/crio-5f398a8c85814878890dd3a72f29687a27824f89850573739161ad755b0173d4 WatchSource:0}: Error finding container 5f398a8c85814878890dd3a72f29687a27824f89850573739161ad755b0173d4: Status 404 returned error can't find the container with id 5f398a8c85814878890dd3a72f29687a27824f89850573739161ad755b0173d4 Nov 03 20:15:01.589760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:01.589713 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:02.465583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.464827 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerStarted","Data":"71bbe0f16126cc1b03c929d4081b830f15cd7e3892cb565dac3251c7d7e482d6"} Nov 03 20:15:02.466789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.466755 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerStarted","Data":"ce0bc19ddf2b9d4a5c64f9fcdc99a60f26a95ca162257e0f85283e510e935df6"} Nov 03 20:15:02.468784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.468752 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerStarted","Data":"a40b930ff7ee133f05610b4cdd3c7c91141dc3b406a1a60ae961a4fe986ed923"} Nov 03 20:15:02.470497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.470470 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerStarted","Data":"81fb9876e5417989c48a45cd5011e436f8eb965acf939eb3be83c5108eacb513"} Nov 03 20:15:02.470618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.470515 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerStarted","Data":"5f398a8c85814878890dd3a72f29687a27824f89850573739161ad755b0173d4"} Nov 03 20:15:02.472070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.472047 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerStarted","Data":"e6396ae3e288bf99dee8a8e6a15cb0cf79d0b3ac7b2d7b7d539e12514c58b128"} Nov 03 20:15:02.490509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.490433 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" podStartSLOduration=2.490421433 podStartE2EDuration="2.490421433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.489475944 +0000 UTC m=+1043.214483541" watchObservedRunningTime="2026-11-03 20:15:02.490421433 +0000 UTC m=+1043.215429056" Nov 03 20:15:02.511401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.511357 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" podStartSLOduration=2.511345074 podStartE2EDuration="2.511345074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.507646891 +0000 UTC m=+1043.232654486" watchObservedRunningTime="2026-11-03 20:15:02.511345074 +0000 UTC m=+1043.236352702" Nov 03 20:15:02.531422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.531380 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" podStartSLOduration=2.531364694 podStartE2EDuration="2.531364694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.525691317 +0000 UTC m=+1043.250698914" watchObservedRunningTime="2026-11-03 20:15:02.531364694 +0000 UTC m=+1043.256372292" Nov 03 20:15:02.546606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.546527 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" podStartSLOduration=2.546514726 podStartE2EDuration="2.546514726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.545367213 +0000 UTC m=+1043.270374812" watchObservedRunningTime="2026-11-03 20:15:02.546514726 +0000 UTC m=+1043.271522320" Nov 03 20:15:02.571412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.571351 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" podStartSLOduration=2.571334231 podStartE2EDuration="2.571334231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.567630137 +0000 UTC m=+1043.292637732" watchObservedRunningTime="2026-11-03 20:15:02.571334231 +0000 UTC m=+1043.296341828" Nov 03 20:15:02.623508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.623462 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:02.797588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.797317 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx"] Nov 03 20:15:02.816126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.810543 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx"] Nov 03 20:15:02.816126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.810669 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.818638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.816783 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:02.856499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.856464 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.856653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.856513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-schpj\" (UniqueName: \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.856753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.856730 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.959493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.959462 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.959656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.959526 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-schpj\" (UniqueName: \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.963283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.959719 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.963283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.960197 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.966770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.966738 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:02.972890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:02.972862 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-schpj\" (UniqueName: \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\") pod \"maas-api-revoke-sub-1ebb2b7f-wkvlx\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:03.252609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.252573 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:03.467984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.467943 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:03.481583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.481550 5954 generic.go:356] "Generic (PLEG): container finished" podID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerID="81fb9876e5417989c48a45cd5011e436f8eb965acf939eb3be83c5108eacb513" exitCode=60 Nov 03 20:15:03.481737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.481598 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerDied","Data":"81fb9876e5417989c48a45cd5011e436f8eb965acf939eb3be83c5108eacb513"} Nov 03 20:15:03.481952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.481935 5954 scope.go:122] "RemoveContainer" containerID="81fb9876e5417989c48a45cd5011e436f8eb965acf939eb3be83c5108eacb513" Nov 03 20:15:03.483349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.483326 5954 generic.go:356] "Generic (PLEG): container finished" podID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerID="e6396ae3e288bf99dee8a8e6a15cb0cf79d0b3ac7b2d7b7d539e12514c58b128" exitCode=60 Nov 03 20:15:03.483474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.483409 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerDied","Data":"e6396ae3e288bf99dee8a8e6a15cb0cf79d0b3ac7b2d7b7d539e12514c58b128"} Nov 03 20:15:03.483631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.483613 5954 scope.go:122] "RemoveContainer" containerID="e6396ae3e288bf99dee8a8e6a15cb0cf79d0b3ac7b2d7b7d539e12514c58b128" Nov 03 20:15:03.485233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.485210 5954 generic.go:356] "Generic (PLEG): container finished" podID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerID="71bbe0f16126cc1b03c929d4081b830f15cd7e3892cb565dac3251c7d7e482d6" exitCode=60 Nov 03 20:15:03.485300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.485231 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerDied","Data":"71bbe0f16126cc1b03c929d4081b830f15cd7e3892cb565dac3251c7d7e482d6"} Nov 03 20:15:03.485896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.485482 5954 scope.go:122] "RemoveContainer" containerID="71bbe0f16126cc1b03c929d4081b830f15cd7e3892cb565dac3251c7d7e482d6" Nov 03 20:15:03.492117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.492087 5954 generic.go:356] "Generic (PLEG): container finished" podID="c4db5328-f561-42ae-bab0-a027349ff03d" containerID="ce0bc19ddf2b9d4a5c64f9fcdc99a60f26a95ca162257e0f85283e510e935df6" exitCode=60 Nov 03 20:15:03.492212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.492154 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerDied","Data":"ce0bc19ddf2b9d4a5c64f9fcdc99a60f26a95ca162257e0f85283e510e935df6"} Nov 03 20:15:03.492444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.492424 5954 scope.go:122] "RemoveContainer" containerID="ce0bc19ddf2b9d4a5c64f9fcdc99a60f26a95ca162257e0f85283e510e935df6" Nov 03 20:15:03.493857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.493838 5954 generic.go:356] "Generic (PLEG): container finished" podID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerID="a40b930ff7ee133f05610b4cdd3c7c91141dc3b406a1a60ae961a4fe986ed923" exitCode=60 Nov 03 20:15:03.493938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.493874 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerDied","Data":"a40b930ff7ee133f05610b4cdd3c7c91141dc3b406a1a60ae961a4fe986ed923"} Nov 03 20:15:03.494120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.494096 5954 scope.go:122] "RemoveContainer" containerID="a40b930ff7ee133f05610b4cdd3c7c91141dc3b406a1a60ae961a4fe986ed923" Nov 03 20:15:03.705909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:03.705879 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx"] Nov 03 20:15:04.201016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.200977 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:04.522232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.522129 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerStarted","Data":"1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34"} Nov 03 20:15:04.526875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.526846 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerStarted","Data":"d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1"} Nov 03 20:15:04.530291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.530264 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerStarted","Data":"6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712"} Nov 03 20:15:04.533427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.533400 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerStarted","Data":"6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47"} Nov 03 20:15:04.535262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.535237 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" event={"ID":"28c4c14f-5110-4813-a35d-fd6d54655558","Type":"ContainerStarted","Data":"6541b08fe31c204e965a285cfe5188c17cdf494e80eee71cebed3fbf138bde66"} Nov 03 20:15:04.535352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.535289 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" event={"ID":"28c4c14f-5110-4813-a35d-fd6d54655558","Type":"ContainerStarted","Data":"fac17e0cc6ff1ac46c8cece0268975a269133dfb1c60f9afd0f787851d727ba2"} Nov 03 20:15:04.537364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.537343 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerStarted","Data":"bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b"} Nov 03 20:15:04.629870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:04.629810 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" podStartSLOduration=2.629796001 podStartE2EDuration="2.629796001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:04.626863564 +0000 UTC m=+1045.351871158" watchObservedRunningTime="2026-11-03 20:15:04.629796001 +0000 UTC m=+1045.354803654" Nov 03 20:15:05.556015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.555139 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:05.614182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.613227 5954 generic.go:356] "Generic (PLEG): container finished" podID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerID="6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47" exitCode=60 Nov 03 20:15:05.614182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.613312 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerDied","Data":"6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47"} Nov 03 20:15:05.614182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.613357 5954 scope.go:122] "RemoveContainer" containerID="81fb9876e5417989c48a45cd5011e436f8eb965acf939eb3be83c5108eacb513" Nov 03 20:15:05.614182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.613801 5954 scope.go:122] "RemoveContainer" containerID="6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47" Nov 03 20:15:05.614537 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:05.614208 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26_odh-ai-gateway-infra(436cd249-ff1f-4b17-b6b2-3592fba16020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" Nov 03 20:15:05.660413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.660373 5954 generic.go:356] "Generic (PLEG): container finished" podID="28c4c14f-5110-4813-a35d-fd6d54655558" containerID="6541b08fe31c204e965a285cfe5188c17cdf494e80eee71cebed3fbf138bde66" exitCode=0 Nov 03 20:15:05.660676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.660649 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" event={"ID":"28c4c14f-5110-4813-a35d-fd6d54655558","Type":"ContainerDied","Data":"6541b08fe31c204e965a285cfe5188c17cdf494e80eee71cebed3fbf138bde66"} Nov 03 20:15:05.666197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.665082 5954 generic.go:356] "Generic (PLEG): container finished" podID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerID="bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b" exitCode=60 Nov 03 20:15:05.666197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.665295 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerDied","Data":"bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b"} Nov 03 20:15:05.666197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.665662 5954 scope.go:122] "RemoveContainer" containerID="bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b" Nov 03 20:15:05.666197 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:05.665960 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt_odh-ai-gateway-infra(440fac0d-03d3-41b9-bf3b-823b357e3c09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" Nov 03 20:15:05.669437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.669413 5954 generic.go:356] "Generic (PLEG): container finished" podID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerID="1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34" exitCode=60 Nov 03 20:15:05.669876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.669478 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerDied","Data":"1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34"} Nov 03 20:15:05.669876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.669691 5954 scope.go:122] "RemoveContainer" containerID="1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34" Nov 03 20:15:05.670059 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:05.669993 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf_odh-ai-gateway-infra(f9638c8b-0de1-4163-b750-7dd3edf6d607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" Nov 03 20:15:05.671698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.671672 5954 generic.go:356] "Generic (PLEG): container finished" podID="c4db5328-f561-42ae-bab0-a027349ff03d" containerID="d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1" exitCode=60 Nov 03 20:15:05.671813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.671754 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerDied","Data":"d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1"} Nov 03 20:15:05.671976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.671958 5954 scope.go:122] "RemoveContainer" containerID="d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1" Nov 03 20:15:05.672360 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:05.672338 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-gs722_odh-ai-gateway-infra(c4db5328-f561-42ae-bab0-a027349ff03d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" Nov 03 20:15:05.673784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.673759 5954 generic.go:356] "Generic (PLEG): container finished" podID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerID="6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712" exitCode=60 Nov 03 20:15:05.673888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.673807 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerDied","Data":"6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712"} Nov 03 20:15:05.674066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.674049 5954 scope.go:122] "RemoveContainer" containerID="6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712" Nov 03 20:15:05.674411 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:05.674392 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7_odh-ai-gateway-infra(3981a3c4-bc4a-43a4-9076-c3fa445f9df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" Nov 03 20:15:05.679671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.679653 5954 scope.go:122] "RemoveContainer" containerID="e6396ae3e288bf99dee8a8e6a15cb0cf79d0b3ac7b2d7b7d539e12514c58b128" Nov 03 20:15:05.693957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.693935 5954 scope.go:122] "RemoveContainer" containerID="71bbe0f16126cc1b03c929d4081b830f15cd7e3892cb565dac3251c7d7e482d6" Nov 03 20:15:05.707558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.707530 5954 scope.go:122] "RemoveContainer" containerID="ce0bc19ddf2b9d4a5c64f9fcdc99a60f26a95ca162257e0f85283e510e935df6" Nov 03 20:15:05.720943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:05.720919 5954 scope.go:122] "RemoveContainer" containerID="a40b930ff7ee133f05610b4cdd3c7c91141dc3b406a1a60ae961a4fe986ed923" Nov 03 20:15:06.539858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.539824 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:06.705374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.705347 5954 scope.go:122] "RemoveContainer" containerID="6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47" Nov 03 20:15:06.705803 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:06.705764 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26_odh-ai-gateway-infra(436cd249-ff1f-4b17-b6b2-3592fba16020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" Nov 03 20:15:06.707069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.707046 5954 scope.go:122] "RemoveContainer" containerID="bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b" Nov 03 20:15:06.707442 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:06.707420 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt_odh-ai-gateway-infra(440fac0d-03d3-41b9-bf3b-823b357e3c09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" Nov 03 20:15:06.708751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.708733 5954 scope.go:122] "RemoveContainer" containerID="1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34" Nov 03 20:15:06.709077 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:06.709053 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf_odh-ai-gateway-infra(f9638c8b-0de1-4163-b750-7dd3edf6d607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" Nov 03 20:15:06.710146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.710127 5954 scope.go:122] "RemoveContainer" containerID="d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1" Nov 03 20:15:06.710416 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:06.710397 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-gs722_odh-ai-gateway-infra(c4db5328-f561-42ae-bab0-a027349ff03d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" Nov 03 20:15:06.711688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.711667 5954 scope.go:122] "RemoveContainer" containerID="6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712" Nov 03 20:15:06.712029 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:06.711929 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7_odh-ai-gateway-infra(3981a3c4-bc4a-43a4-9076-c3fa445f9df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" Nov 03 20:15:06.882326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.882302 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:06.987830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.987799 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\") pod \"28c4c14f-5110-4813-a35d-fd6d54655558\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " Nov 03 20:15:06.988021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.987906 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\" (UniqueName: \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\") pod \"28c4c14f-5110-4813-a35d-fd6d54655558\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " Nov 03 20:15:06.988021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.987955 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\") pod \"28c4c14f-5110-4813-a35d-fd6d54655558\" (UID: \"28c4c14f-5110-4813-a35d-fd6d54655558\") " Nov 03 20:15:06.988369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.988343 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca" pod "28c4c14f-5110-4813-a35d-fd6d54655558" (UID: "28c4c14f-5110-4813-a35d-fd6d54655558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:06.988780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.988758 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:06.990003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.989975 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj" pod "28c4c14f-5110-4813-a35d-fd6d54655558" (UID: "28c4c14f-5110-4813-a35d-fd6d54655558"). InnerVolumeSpecName "kube-api-access-schpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:06.990003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:06.989983 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca" pod "28c4c14f-5110-4813-a35d-fd6d54655558" (UID: "28c4c14f-5110-4813-a35d-fd6d54655558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:07.090594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.090561 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c4c14f-5110-4813-a35d-fd6d54655558-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:07.090594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.090592 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-schpj\" (UniqueName: \"kubernetes.io/projected/28c4c14f-5110-4813-a35d-fd6d54655558-kube-api-access-schpj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:07.407613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.407577 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:07.722009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.721914 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" Nov 03 20:15:07.722464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.722027 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx" event={"ID":"28c4c14f-5110-4813-a35d-fd6d54655558","Type":"ContainerDied","Data":"fac17e0cc6ff1ac46c8cece0268975a269133dfb1c60f9afd0f787851d727ba2"} Nov 03 20:15:07.722464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:07.722082 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac17e0cc6ff1ac46c8cece0268975a269133dfb1c60f9afd0f787851d727ba2" Nov 03 20:15:08.180287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:08.180252 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:09.175946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:09.175914 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:09.974645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:09.974611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:10.919120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:10.919087 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:11.714268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:11.714233 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:12.548034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:12.548004 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:13.409093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:13.409051 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:14.446132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:14.432855 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:15.398302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.398263 5954 scope.go:122] "RemoveContainer" containerID="6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47" Nov 03 20:15:15.398498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.398438 5954 scope.go:122] "RemoveContainer" containerID="1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34" Nov 03 20:15:15.398602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.398581 5954 scope.go:122] "RemoveContainer" containerID="bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b" Nov 03 20:15:15.454789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.454764 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:15.875594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.875562 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerStarted","Data":"27f4ed2f83610f353378e15eeada0faae8fb3fd7104687879dc9c1825ddff572"} Nov 03 20:15:15.877643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.877621 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerStarted","Data":"88e9df7b5302128b57623e1ae327dbe41fb0daab1c3bfbb5b0b960ead20d0332"} Nov 03 20:15:15.879555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:15.879535 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerStarted","Data":"e81cb95a39bbc64e9fa43c784111dac003dec978adc6f6dc090cca4919ce553d"} Nov 03 20:15:16.262696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.262655 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:15:16.263550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.263525 5954 cpu_manager.go:414] "containerMap: removing container" podUID="28c4c14f-5110-4813-a35d-fd6d54655558" containerName="revoke-keys" Nov 03 20:15:16.263550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.263548 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c4c14f-5110-4813-a35d-fd6d54655558" containerName="revoke-keys" Nov 03 20:15:16.263807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.263793 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c4c14f-5110-4813-a35d-fd6d54655558" containerName="revoke-keys" Nov 03 20:15:16.268642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.268609 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.292708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.292669 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:15:16.405497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405461 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405511 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405542 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405605 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c47r\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405787 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405816 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.405854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405837 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.406001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.405986 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.406044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.406031 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507315 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507360 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507391 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c47r\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507613 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507641 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507667 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.507815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507775 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507837 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.507904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.508016 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.508093 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.508230 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.508688 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.508824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.508743 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.510945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.510920 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.511097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.511083 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.541485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.541417 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.542027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.542008 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c47r\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.586254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.586229 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:16.611817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.611790 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:16.892541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.892513 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:15:16.892730 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:16.892570 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a41eb9_59df_42a4_81eb_4413e5b204a2.slice/crio-fd4aa2f96a7dba5805f46b29a843a0bc1df946daebae6144a2efc74651f22005 WatchSource:0}: Error finding container fd4aa2f96a7dba5805f46b29a843a0bc1df946daebae6144a2efc74651f22005: Status 404 returned error can't find the container with id fd4aa2f96a7dba5805f46b29a843a0bc1df946daebae6144a2efc74651f22005 Nov 03 20:15:16.893884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.893864 5954 generic.go:356] "Generic (PLEG): container finished" podID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerID="88e9df7b5302128b57623e1ae327dbe41fb0daab1c3bfbb5b0b960ead20d0332" exitCode=60 Nov 03 20:15:16.893970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.893913 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerDied","Data":"88e9df7b5302128b57623e1ae327dbe41fb0daab1c3bfbb5b0b960ead20d0332"} Nov 03 20:15:16.894029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.893969 5954 scope.go:122] "RemoveContainer" containerID="6f9ad80391d5037f7a4cadb52c8803ceec0cc3bb5369c43247f77cf0c0aadf47" Nov 03 20:15:16.894378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.894360 5954 scope.go:122] "RemoveContainer" containerID="88e9df7b5302128b57623e1ae327dbe41fb0daab1c3bfbb5b0b960ead20d0332" Nov 03 20:15:16.894830 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:16.894795 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26_odh-ai-gateway-infra(436cd249-ff1f-4b17-b6b2-3592fba16020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" Nov 03 20:15:16.895694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.895668 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:16.895767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.895738 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:16.895819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.895785 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:16.896679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.896657 5954 generic.go:356] "Generic (PLEG): container finished" podID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerID="e81cb95a39bbc64e9fa43c784111dac003dec978adc6f6dc090cca4919ce553d" exitCode=60 Nov 03 20:15:16.896777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.896723 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerDied","Data":"e81cb95a39bbc64e9fa43c784111dac003dec978adc6f6dc090cca4919ce553d"} Nov 03 20:15:16.897076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.897058 5954 scope.go:122] "RemoveContainer" containerID="e81cb95a39bbc64e9fa43c784111dac003dec978adc6f6dc090cca4919ce553d" Nov 03 20:15:16.898325 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:16.898297 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt_odh-ai-gateway-infra(440fac0d-03d3-41b9-bf3b-823b357e3c09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" Nov 03 20:15:16.899233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.899208 5954 generic.go:356] "Generic (PLEG): container finished" podID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerID="27f4ed2f83610f353378e15eeada0faae8fb3fd7104687879dc9c1825ddff572" exitCode=60 Nov 03 20:15:16.899320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.899276 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerDied","Data":"27f4ed2f83610f353378e15eeada0faae8fb3fd7104687879dc9c1825ddff572"} Nov 03 20:15:16.899683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.899662 5954 scope.go:122] "RemoveContainer" containerID="27f4ed2f83610f353378e15eeada0faae8fb3fd7104687879dc9c1825ddff572" Nov 03 20:15:16.899973 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:16.899929 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf_odh-ai-gateway-infra(f9638c8b-0de1-4163-b750-7dd3edf6d607)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" Nov 03 20:15:16.910364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.910347 5954 scope.go:122] "RemoveContainer" containerID="bd7312971aef24c2c9b4c1e53a54753542eb2c3bab3d394f79df9c840e6aff6b" Nov 03 20:15:16.925633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.925617 5954 scope.go:122] "RemoveContainer" containerID="1cd168f723ce1b62b1445fdabf32479c49a644c6aad8e0634d99078bea809e34" Nov 03 20:15:16.929201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.929177 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:16.938407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.938384 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:16.979816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:16.979787 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:17.086836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.086793 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq"] Nov 03 20:15:17.091952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.091924 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.097804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.097781 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:17.106155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.105311 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq"] Nov 03 20:15:17.131617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.131588 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.131891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.131870 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.132029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.132007 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9d7\" (UniqueName: \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.233728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.233641 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.233895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.233732 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9d7\" (UniqueName: \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.233971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.233913 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.234406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.234385 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.236530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.236508 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.246600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.246578 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9d7\" (UniqueName: \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\") pod \"maas-api-revoke-sub-50a42809-fqfjq\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.412613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.412581 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:17.712621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.712589 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:17.773632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:17.755727 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq"] Nov 03 20:15:18.000835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.000801 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" event={"ID":"16a41eb9-59df-42a4-81eb-4413e5b204a2","Type":"ContainerStarted","Data":"70440dbd7097774ad601a92b3b551329a90fa299c269abe4688a46f689a2c020"} Nov 03 20:15:18.000948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.000908 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" event={"ID":"16a41eb9-59df-42a4-81eb-4413e5b204a2","Type":"ContainerStarted","Data":"fd4aa2f96a7dba5805f46b29a843a0bc1df946daebae6144a2efc74651f22005"} Nov 03 20:15:18.002985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.002955 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" event={"ID":"9269629c-101c-45e0-bda0-35965748e13e","Type":"ContainerStarted","Data":"ff048d10546bd456b646ab59761fc2ab2baeeaa24da1291202a2a78d0e128158"} Nov 03 20:15:18.091800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.091693 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podStartSLOduration=2.091679185 podStartE2EDuration="2.091679185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:18.089370285 +0000 UTC m=+1058.814377881" watchObservedRunningTime="2026-11-03 20:15:18.091679185 +0000 UTC m=+1058.816686781" Nov 03 20:15:18.281720 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.281690 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:18.348091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.347744 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:18.376509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.376476 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\") pod \"f9638c8b-0de1-4163-b750-7dd3edf6d607\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " Nov 03 20:15:18.376691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.376549 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\" (UniqueName: \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\") pod \"440fac0d-03d3-41b9-bf3b-823b357e3c09\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " Nov 03 20:15:18.376691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.376686 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\") pod \"440fac0d-03d3-41b9-bf3b-823b357e3c09\" (UID: \"440fac0d-03d3-41b9-bf3b-823b357e3c09\") " Nov 03 20:15:18.376816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.376717 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\" (UniqueName: \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\") pod \"f9638c8b-0de1-4163-b750-7dd3edf6d607\" (UID: \"f9638c8b-0de1-4163-b750-7dd3edf6d607\") " Nov 03 20:15:18.377680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.377649 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca" pod "440fac0d-03d3-41b9-bf3b-823b357e3c09" (UID: "440fac0d-03d3-41b9-bf3b-823b357e3c09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:18.378176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.378154 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca" pod "f9638c8b-0de1-4163-b750-7dd3edf6d607" (UID: "f9638c8b-0de1-4163-b750-7dd3edf6d607"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:18.383623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.383597 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt" pod "440fac0d-03d3-41b9-bf3b-823b357e3c09" (UID: "440fac0d-03d3-41b9-bf3b-823b357e3c09"). InnerVolumeSpecName "kube-api-access-nz2dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:18.386131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.385839 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc" pod "f9638c8b-0de1-4163-b750-7dd3edf6d607" (UID: "f9638c8b-0de1-4163-b750-7dd3edf6d607"). InnerVolumeSpecName "kube-api-access-rk9lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:18.397863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.397840 5954 scope.go:122] "RemoveContainer" containerID="d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1" Nov 03 20:15:18.409448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.409431 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:18.478675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.478644 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\" (UniqueName: \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\") pod \"436cd249-ff1f-4b17-b6b2-3592fba16020\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " Nov 03 20:15:18.478834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.478783 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\") pod \"436cd249-ff1f-4b17-b6b2-3592fba16020\" (UID: \"436cd249-ff1f-4b17-b6b2-3592fba16020\") " Nov 03 20:15:18.479474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.479450 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz2dt\" (UniqueName: \"kubernetes.io/projected/440fac0d-03d3-41b9-bf3b-823b357e3c09-kube-api-access-nz2dt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.479575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.479479 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/440fac0d-03d3-41b9-bf3b-823b357e3c09-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.479575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.479494 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk9lc\" (UniqueName: \"kubernetes.io/projected/f9638c8b-0de1-4163-b750-7dd3edf6d607-kube-api-access-rk9lc\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.479575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.479508 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9638c8b-0de1-4163-b750-7dd3edf6d607-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.479855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.479835 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca" pod "436cd249-ff1f-4b17-b6b2-3592fba16020" (UID: "436cd249-ff1f-4b17-b6b2-3592fba16020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:18.481827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.481793 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg" pod "436cd249-ff1f-4b17-b6b2-3592fba16020" (UID: "436cd249-ff1f-4b17-b6b2-3592fba16020"). InnerVolumeSpecName "kube-api-access-x65rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:18.580937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.580904 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65rg\" (UniqueName: \"kubernetes.io/projected/436cd249-ff1f-4b17-b6b2-3592fba16020-kube-api-access-x65rg\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.580937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.580933 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/436cd249-ff1f-4b17-b6b2-3592fba16020-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.586709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.586685 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:18.588006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.587985 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:18.588095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.588032 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:18.667513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:18.667479 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:19.045860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.045835 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" Nov 03 20:15:19.046219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.045865 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26" event={"ID":"436cd249-ff1f-4b17-b6b2-3592fba16020","Type":"ContainerDied","Data":"5f398a8c85814878890dd3a72f29687a27824f89850573739161ad755b0173d4"} Nov 03 20:15:19.046219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.045928 5954 scope.go:122] "RemoveContainer" containerID="88e9df7b5302128b57623e1ae327dbe41fb0daab1c3bfbb5b0b960ead20d0332" Nov 03 20:15:19.047793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.047764 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" event={"ID":"440fac0d-03d3-41b9-bf3b-823b357e3c09","Type":"ContainerDied","Data":"b62549525eb65e90f43cb50ab2f98c3d2c963a290a27fffe19b8d854f770897e"} Nov 03 20:15:19.047899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.047878 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt" Nov 03 20:15:19.054922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.054906 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" Nov 03 20:15:19.055032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.054952 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf" event={"ID":"f9638c8b-0de1-4163-b750-7dd3edf6d607","Type":"ContainerDied","Data":"1accb08d01691d0f1bfabc2508a443dc655eadc9f3edd96364f1864010507522"} Nov 03 20:15:19.057029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.057004 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" event={"ID":"9269629c-101c-45e0-bda0-35965748e13e","Type":"ContainerStarted","Data":"5d2fc9bf413dbdcb0dce4b849c874026f0416adfcc60ef7885808d7cdd41d62b"} Nov 03 20:15:19.065981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.065961 5954 scope.go:122] "RemoveContainer" containerID="e81cb95a39bbc64e9fa43c784111dac003dec978adc6f6dc090cca4919ce553d" Nov 03 20:15:19.075100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.075083 5954 scope.go:122] "RemoveContainer" containerID="27f4ed2f83610f353378e15eeada0faae8fb3fd7104687879dc9c1825ddff572" Nov 03 20:15:19.105740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.105710 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:19.110506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.110481 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895615-68q26"] Nov 03 20:15:19.144604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.144575 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:19.149178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.149154 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a6f98b-29895615-xpkpf"] Nov 03 20:15:19.157268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.157245 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:19.161596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.161575 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895615-sdcbt"] Nov 03 20:15:19.405861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.405830 5954 scope.go:122] "RemoveContainer" containerID="6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712" Nov 03 20:15:19.411283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.411259 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" path="/var/lib/kubelet/pods/436cd249-ff1f-4b17-b6b2-3592fba16020/volumes" Nov 03 20:15:19.411826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.411804 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" path="/var/lib/kubelet/pods/440fac0d-03d3-41b9-bf3b-823b357e3c09/volumes" Nov 03 20:15:19.412394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.412374 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" path="/var/lib/kubelet/pods/f9638c8b-0de1-4163-b750-7dd3edf6d607/volumes" Nov 03 20:15:19.587007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.586978 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:19.587182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.587025 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:19.690236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:19.690141 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:20.068460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.068430 5954 generic.go:356] "Generic (PLEG): container finished" podID="c4db5328-f561-42ae-bab0-a027349ff03d" containerID="4fddb93f3abd6eafa64702121d1ba9808cbde1eead5246a4ceaa311e852afb3e" exitCode=60 Nov 03 20:15:20.068928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.068505 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerDied","Data":"4fddb93f3abd6eafa64702121d1ba9808cbde1eead5246a4ceaa311e852afb3e"} Nov 03 20:15:20.068928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.068560 5954 scope.go:122] "RemoveContainer" containerID="d1d59c1f48bcfca4f3425d426ebec6cc156469dbc1ec56a8ace1bebf659dbaf1" Nov 03 20:15:20.069007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.068971 5954 scope.go:122] "RemoveContainer" containerID="4fddb93f3abd6eafa64702121d1ba9808cbde1eead5246a4ceaa311e852afb3e" Nov 03 20:15:20.069415 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:20.069390 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-gs722_odh-ai-gateway-infra(c4db5328-f561-42ae-bab0-a027349ff03d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" Nov 03 20:15:20.070767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.070746 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerStarted","Data":"aee709da4906fed632d0aceb352ab285480439a088f282764cf8a7f530d130e4"} Nov 03 20:15:20.072277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.072256 5954 generic.go:356] "Generic (PLEG): container finished" podID="9269629c-101c-45e0-bda0-35965748e13e" containerID="5d2fc9bf413dbdcb0dce4b849c874026f0416adfcc60ef7885808d7cdd41d62b" exitCode=0 Nov 03 20:15:20.072359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.072341 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" event={"ID":"9269629c-101c-45e0-bda0-35965748e13e","Type":"ContainerDied","Data":"5d2fc9bf413dbdcb0dce4b849c874026f0416adfcc60ef7885808d7cdd41d62b"} Nov 03 20:15:20.299395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.299367 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:20.368100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.368066 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\" (UniqueName: \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\") pod \"9269629c-101c-45e0-bda0-35965748e13e\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " Nov 03 20:15:20.368433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.368415 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\") pod \"9269629c-101c-45e0-bda0-35965748e13e\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " Nov 03 20:15:20.368513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.368473 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\") pod \"9269629c-101c-45e0-bda0-35965748e13e\" (UID: \"9269629c-101c-45e0-bda0-35965748e13e\") " Nov 03 20:15:20.368902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.368872 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca" pod "9269629c-101c-45e0-bda0-35965748e13e" (UID: "9269629c-101c-45e0-bda0-35965748e13e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.369327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.369304 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9269629c-101c-45e0-bda0-35965748e13e-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.370484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.370457 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7" pod "9269629c-101c-45e0-bda0-35965748e13e" (UID: "9269629c-101c-45e0-bda0-35965748e13e"). InnerVolumeSpecName "kube-api-access-th9d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.370613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.370510 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca" pod "9269629c-101c-45e0-bda0-35965748e13e" (UID: "9269629c-101c-45e0-bda0-35965748e13e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:20.470723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.470690 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9269629c-101c-45e0-bda0-35965748e13e-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.470723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.470718 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th9d7\" (UniqueName: \"kubernetes.io/projected/9269629c-101c-45e0-bda0-35965748e13e-kube-api-access-th9d7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.587501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.587425 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:20.587501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.587486 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:20.710133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:20.710081 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:21.088270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.088235 5954 generic.go:356] "Generic (PLEG): container finished" podID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerID="aee709da4906fed632d0aceb352ab285480439a088f282764cf8a7f530d130e4" exitCode=60 Nov 03 20:15:21.088744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.088310 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerDied","Data":"aee709da4906fed632d0aceb352ab285480439a088f282764cf8a7f530d130e4"} Nov 03 20:15:21.088744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.088406 5954 scope.go:122] "RemoveContainer" containerID="6d1eac5ff474d9455835f155bdd98f59d97669861b6fef0847c7d65f8ce4f712" Nov 03 20:15:21.088861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.088843 5954 scope.go:122] "RemoveContainer" containerID="aee709da4906fed632d0aceb352ab285480439a088f282764cf8a7f530d130e4" Nov 03 20:15:21.089311 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:21.089285 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7_odh-ai-gateway-infra(3981a3c4-bc4a-43a4-9076-c3fa445f9df6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" Nov 03 20:15:21.089997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.089981 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" Nov 03 20:15:21.090052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.090021 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq" event={"ID":"9269629c-101c-45e0-bda0-35965748e13e","Type":"ContainerDied","Data":"ff048d10546bd456b646ab59761fc2ab2baeeaa24da1291202a2a78d0e128158"} Nov 03 20:15:21.090091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.090067 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff048d10546bd456b646ab59761fc2ab2baeeaa24da1291202a2a78d0e128158" Nov 03 20:15:21.104798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.104778 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:21.151489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.151448 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:21.354264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.354238 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:21.509556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.509527 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\" (UniqueName: \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\") pod \"c4db5328-f561-42ae-bab0-a027349ff03d\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " Nov 03 20:15:21.509723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.509686 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\") pod \"c4db5328-f561-42ae-bab0-a027349ff03d\" (UID: \"c4db5328-f561-42ae-bab0-a027349ff03d\") " Nov 03 20:15:21.509977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.509955 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca" pod "c4db5328-f561-42ae-bab0-a027349ff03d" (UID: "c4db5328-f561-42ae-bab0-a027349ff03d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:21.510508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.510488 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4db5328-f561-42ae-bab0-a027349ff03d-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.511733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.511715 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m" pod "c4db5328-f561-42ae-bab0-a027349ff03d" (UID: "c4db5328-f561-42ae-bab0-a027349ff03d"). InnerVolumeSpecName "kube-api-access-4pd7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:21.587380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.587342 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:21.587535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.587406 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:21.611975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.611893 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pd7m\" (UniqueName: \"kubernetes.io/projected/c4db5328-f561-42ae-bab0-a027349ff03d-kube-api-access-4pd7m\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.644880 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk"] Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645650 5954 cpu_manager.go:414] "containerMap: removing container" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645668 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645691 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645696 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645707 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645713 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645722 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645726 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645741 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645746 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645752 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9269629c-101c-45e0-bda0-35965748e13e" containerName="revoke-keys" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645756 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9269629c-101c-45e0-bda0-35965748e13e" containerName="revoke-keys" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645763 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645767 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645777 5954 cpu_manager.go:414] "containerMap: removing container" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645781 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645800 5954 cpu_manager.go:414] "containerMap: removing container" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645804 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645823 5954 cpu_manager.go:414] "containerMap: removing container" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645829 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645936 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645947 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645954 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645963 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645973 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645981 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645988 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.645995 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.646001 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.646007 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.646013 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9638c8b-0de1-4163-b750-7dd3edf6d607" containerName="cleanup" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.646018 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9269629c-101c-45e0-bda0-35965748e13e" containerName="revoke-keys" Nov 03 20:15:21.646482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.646022 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:21.650225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.650184 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.655342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.654740 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:21.664224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.664200 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk"] Nov 03 20:15:21.742038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.741998 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:21.852134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.847201 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.852134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.847423 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2z5t\" (UniqueName: \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.852134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.847505 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.949507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.949424 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.949648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.949520 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2z5t\" (UniqueName: \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.949648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.949558 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.950386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.950360 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.952144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.952001 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.962471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.962450 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2z5t\" (UniqueName: \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\") pod \"maas-api-revoke-sub-bf0721ec-gfkvk\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:21.969434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:21.969413 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:22.104826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.104783 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" event={"ID":"c4db5328-f561-42ae-bab0-a027349ff03d","Type":"ContainerDied","Data":"2cf872c7b7872051eb35232cd3206fa8e4c35f1322435797f45fc3b0e55d357f"} Nov 03 20:15:22.105308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.104863 5954 scope.go:122] "RemoveContainer" containerID="4fddb93f3abd6eafa64702121d1ba9808cbde1eead5246a4ceaa311e852afb3e" Nov 03 20:15:22.105383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.105359 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722" Nov 03 20:15:22.198726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.198681 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:22.206436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.206357 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-gs722"] Nov 03 20:15:22.281706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.281676 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk"] Nov 03 20:15:22.281942 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:22.281747 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f9e244_296d_4e4f_ba75_3fd0b3717d46.slice/crio-26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc WatchSource:0}: Error finding container 26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc: Status 404 returned error can't find the container with id 26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc Nov 03 20:15:22.351387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.351356 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:22.459664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.459578 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\") pod \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " Nov 03 20:15:22.459995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.459971 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca" pod "3981a3c4-bc4a-43a4-9076-c3fa445f9df6" (UID: "3981a3c4-bc4a-43a4-9076-c3fa445f9df6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:22.460217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.460195 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\" (UniqueName: \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\") pod \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\" (UID: \"3981a3c4-bc4a-43a4-9076-c3fa445f9df6\") " Nov 03 20:15:22.461135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.461096 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.541922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.541878 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp" pod "3981a3c4-bc4a-43a4-9076-c3fa445f9df6" (UID: "3981a3c4-bc4a-43a4-9076-c3fa445f9df6"). InnerVolumeSpecName "kube-api-access-jwgtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:22.563829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.563766 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgtp\" (UniqueName: \"kubernetes.io/projected/3981a3c4-bc4a-43a4-9076-c3fa445f9df6-kube-api-access-jwgtp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:22.586586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.586559 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:22.586688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.586613 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:22.635842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:22.635683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:23.122196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.122165 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" Nov 03 20:15:23.122196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.122172 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7" event={"ID":"3981a3c4-bc4a-43a4-9076-c3fa445f9df6","Type":"ContainerDied","Data":"37b92719483fe49261d465ba301f5a2a01370faa397dd2c9a0a93dbba53d6f76"} Nov 03 20:15:23.122710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.122242 5954 scope.go:122] "RemoveContainer" containerID="aee709da4906fed632d0aceb352ab285480439a088f282764cf8a7f530d130e4" Nov 03 20:15:23.124257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.124225 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" event={"ID":"f0f9e244-296d-4e4f-ba75-3fd0b3717d46","Type":"ContainerStarted","Data":"1335d8d1f7d04a5034124c07249f4126a9db4c3905fc478ea2404eccb33086e2"} Nov 03 20:15:23.124388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.124285 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" event={"ID":"f0f9e244-296d-4e4f-ba75-3fd0b3717d46","Type":"ContainerStarted","Data":"26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc"} Nov 03 20:15:23.147952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.147892 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" podStartSLOduration=2.147872866 podStartE2EDuration="2.147872866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:23.144976528 +0000 UTC m=+1063.869984148" watchObservedRunningTime="2026-11-03 20:15:23.147872866 +0000 UTC m=+1063.872880467" Nov 03 20:15:23.162483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.162450 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:23.169829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.169800 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3aba31-29895615-bkmd7"] Nov 03 20:15:23.194404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.194372 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:15:23.407145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.407034 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" path="/var/lib/kubelet/pods/3981a3c4-bc4a-43a4-9076-c3fa445f9df6/volumes" Nov 03 20:15:23.407561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.407546 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" path="/var/lib/kubelet/pods/c4db5328-f561-42ae-bab0-a027349ff03d/volumes" Nov 03 20:15:23.446358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.446322 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:23.588345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.587504 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 20:15:23.588345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:23.587582 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 20:15:24.172133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:24.166241 5954 generic.go:356] "Generic (PLEG): container finished" podID="f0f9e244-296d-4e4f-ba75-3fd0b3717d46" containerID="1335d8d1f7d04a5034124c07249f4126a9db4c3905fc478ea2404eccb33086e2" exitCode=0 Nov 03 20:15:24.172133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:24.166418 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" event={"ID":"f0f9e244-296d-4e4f-ba75-3fd0b3717d46","Type":"ContainerDied","Data":"1335d8d1f7d04a5034124c07249f4126a9db4c3905fc478ea2404eccb33086e2"} Nov 03 20:15:25.353872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.353626 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.497407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.497369 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:25.524014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.523971 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.644873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.644221 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:25.644873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.644304 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.645858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.645343 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.645858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:25.645628 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:26.145360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.145322 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:26.288463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.288422 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:15:26.557197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.542506 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:26.557197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.542900 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:26.557197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.543181 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:15:26.557197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.543944 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:15:26.557197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.543999 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:15:26.632791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.621382 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:15:26.653069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.649199 5954 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:26.653069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.649281 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.776182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.686353 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.776182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.686713 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:26.776182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.687028 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:15:26.776182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.687088 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:15:26.883160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.882210 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:26.883665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:26.883612 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:27.051716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.042999 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:15:27.240316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.240207 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 03 20:15:27.516237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.516123 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:15:27.553838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.553786 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.580133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.554065 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:27.580133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.554116 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:27.580133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.567868 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:27.710132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.704213 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.733132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.732792 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.766133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.747212 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.766133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.753308 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:15:27.766133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.754546 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:15:27.792188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.762668 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:15:27.894351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:27.894313 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\" (UniqueName: \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\") pod \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " Nov 03 20:15:28.649143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.648769 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\") pod \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " Nov 03 20:15:28.649603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.649047 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\") pod \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\" (UID: \"f0f9e244-296d-4e4f-ba75-3fd0b3717d46\") " Nov 03 20:15:28.681134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.659159 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t" pod "f0f9e244-296d-4e4f-ba75-3fd0b3717d46" (UID: "f0f9e244-296d-4e4f-ba75-3fd0b3717d46"). InnerVolumeSpecName "kube-api-access-t2z5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:28.783193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.783066 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:15:28.948315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.940904 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.017261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.593760 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:15:29.063263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.014308 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:15:29.077216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:28.595365 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 03 20:15:29.081212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.058405 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca" pod "f0f9e244-296d-4e4f-ba75-3fd0b3717d46" (UID: "f0f9e244-296d-4e4f-ba75-3fd0b3717d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:29.124550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.124495 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:29.162447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.154061 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:29.188395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.181746 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:29.719144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.716993 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:29.853385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.853338 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2z5t\" (UniqueName: \"kubernetes.io/projected/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-kube-api-access-t2z5t\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.000438 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.878290 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.878062 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.927606 5954 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.015312 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.022255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.714490 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl" podUID="930811f6-8cc8-4139-900f-7b4383999138" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.026943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:29.926542 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:30.097257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.089956 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:30.097563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.097517 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.097710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.090168 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 20:15:30.097858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.090098 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded" Nov 03 20:15:30.103204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.096064 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:30.103460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.103412 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.147234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.115265 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca" pod "f0f9e244-296d-4e4f-ba75-3fd0b3717d46" (UID: "f0f9e244-296d-4e4f-ba75-3fd0b3717d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:30.166497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.166443 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.201647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.184408 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:30.201647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.185828 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.205148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.205090 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:15:30.311851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.239961 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f9e244-296d-4e4f-ba75-3fd0b3717d46-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.791136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.680282 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" Nov 03 20:15:30.794629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.792862 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:30.794629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.792925 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.812963 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:30.812756 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.388s" Nov 03 20:15:30.897695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.896529 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk" event={"ID":"f0f9e244-296d-4e4f-ba75-3fd0b3717d46","Type":"ContainerDied","Data":"26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc"} Nov 03 20:15:30.897695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.897330 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26afd9b7cfd7e2978c84a25dfa47865bf296d42a0be5c47ec40f994879ec67fc" Nov 03 20:15:30.897695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:30.880616 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Nov 03 20:15:31.115192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.099139 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.174430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.117251 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:15:31.174430 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:15:31.150582 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:15:31.174430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.171997 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.217132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.214591 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:31.217468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.217437 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:15:31.218167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.218140 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:31.367600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.367503 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.711085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:31.710996 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.092179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.086668 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:15:32.108177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.101166 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:15:32.108177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.104782 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:15:32.125431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.125403 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.127783 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf"] Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.128906 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.128926 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.128968 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.128976 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.129001 5954 cpu_manager.go:414] "containerMap: removing container" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:32.133941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.129010 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440fac0d-03d3-41b9-bf3b-823b357e3c09" containerName="cleanup" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152548 5954 cpu_manager.go:414] "containerMap: removing container" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152573 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436cd249-ff1f-4b17-b6b2-3592fba16020" containerName="cleanup" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152833 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152844 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db5328-f561-42ae-bab0-a027349ff03d" containerName="cleanup" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152867 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f0f9e244-296d-4e4f-ba75-3fd0b3717d46" containerName="revoke-keys" Nov 03 20:15:32.153281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.152875 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f9e244-296d-4e4f-ba75-3fd0b3717d46" containerName="revoke-keys" Nov 03 20:15:32.154239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.154217 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.155161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.154857 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.155161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.154919 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f9e244-296d-4e4f-ba75-3fd0b3717d46" containerName="revoke-keys" Nov 03 20:15:32.155161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.154937 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:32.176963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.174187 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:32.187342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.187316 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf"] Nov 03 20:15:32.187516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.187417 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:15:32.188294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.188047 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.197618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.197594 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:32.199515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.199489 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:15:32.225057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.224157 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:32.239128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.232317 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:32.239128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.235337 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.239128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.235525 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.239128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.235582 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.247756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.247734 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:15:32.252719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.252655 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:15:32.337487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.337455 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.337659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.337537 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.337659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.337558 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.338561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.338532 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.340652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.340618 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.357001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.354634 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\") pod \"maas-api-revoke-sub-b3be034e-8xjwf\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.565439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.565405 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:32.654650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.651975 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.878180 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:32.878150 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1396f0_a7c9_4c84_b169_a359a01fc45b.slice/crio-fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406 WatchSource:0}: Error finding container fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406: Status 404 returned error can't find the container with id fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406 Nov 03 20:15:32.879048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:32.878969 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf"] Nov 03 20:15:33.103500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.103467 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" event={"ID":"4d1396f0-a7c9-4c84-b169-a359a01fc45b","Type":"ContainerStarted","Data":"fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406"} Nov 03 20:15:33.481759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.481678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.663825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.663790 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz"] Nov 03 20:15:33.664712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.664688 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:33.664712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.664710 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3981a3c4-bc4a-43a4-9076-c3fa445f9df6" containerName="cleanup" Nov 03 20:15:33.668895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.668876 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.677040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.677014 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz"] Nov 03 20:15:33.771258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.771179 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdvr\" (UniqueName: \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.771413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.771295 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.771413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.771343 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.872941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.872904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdvr\" (UniqueName: \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.873070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.873018 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.873118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.873064 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.873793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.873772 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.875325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.875307 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.885078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.885054 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdvr\" (UniqueName: \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\") pod \"maas-api-revoke-sub-41d22c1a-mcdxz\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:33.990426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:33.990390 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:34.116433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:34.116402 5954 generic.go:356] "Generic (PLEG): container finished" podID="4d1396f0-a7c9-4c84-b169-a359a01fc45b" containerID="87dca1abe99215a99557a310eb794aa820b98a02487cd1582af6b9aaadb5812a" exitCode=0 Nov 03 20:15:34.116930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:34.116488 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" event={"ID":"4d1396f0-a7c9-4c84-b169-a359a01fc45b","Type":"ContainerDied","Data":"87dca1abe99215a99557a310eb794aa820b98a02487cd1582af6b9aaadb5812a"} Nov 03 20:15:34.276437 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:34.276399 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e0b48e_8d31_489e_8759_281e75132e4e.slice/crio-b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39 WatchSource:0}: Error finding container b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39: Status 404 returned error can't find the container with id b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39 Nov 03 20:15:34.277331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:34.277304 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz"] Nov 03 20:15:34.400968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:34.400938 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.128018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.127977 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" event={"ID":"e3e0b48e-8d31-489e-8759-281e75132e4e","Type":"ContainerStarted","Data":"1200604f83c13c671cfc1cf0c304a3866e44a54198d025b389882c3fbd523a87"} Nov 03 20:15:35.128474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.128038 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" event={"ID":"e3e0b48e-8d31-489e-8759-281e75132e4e","Type":"ContainerStarted","Data":"b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39"} Nov 03 20:15:35.155848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.155786 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" podStartSLOduration=2.155770843 podStartE2EDuration="2.155770843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:35.150381564 +0000 UTC m=+1075.875389161" watchObservedRunningTime="2026-11-03 20:15:35.155770843 +0000 UTC m=+1075.880778437" Nov 03 20:15:35.205694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.205657 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.426289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.425428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.440439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.440402 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:35.529029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.528983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.621729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.621696 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\") pod \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " Nov 03 20:15:35.621930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.621798 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\") pod \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " Nov 03 20:15:35.621930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.621864 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\") pod \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\" (UID: \"4d1396f0-a7c9-4c84-b169-a359a01fc45b\") " Nov 03 20:15:35.622144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.622100 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca" pod "4d1396f0-a7c9-4c84-b169-a359a01fc45b" (UID: "4d1396f0-a7c9-4c84-b169-a359a01fc45b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:35.622522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.622500 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:35.623926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.623901 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca" pod "4d1396f0-a7c9-4c84-b169-a359a01fc45b" (UID: "4d1396f0-a7c9-4c84-b169-a359a01fc45b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:35.624028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.623959 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm" pod "4d1396f0-a7c9-4c84-b169-a359a01fc45b" (UID: "4d1396f0-a7c9-4c84-b169-a359a01fc45b"). InnerVolumeSpecName "kube-api-access-bkzvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:35.723910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.723877 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzvm\" (UniqueName: \"kubernetes.io/projected/4d1396f0-a7c9-4c84-b169-a359a01fc45b-kube-api-access-bkzvm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:35.723910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:35.723908 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1396f0-a7c9-4c84-b169-a359a01fc45b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:36.092235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.092142 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:36.266711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.253198 5954 generic.go:356] "Generic (PLEG): container finished" podID="e3e0b48e-8d31-489e-8759-281e75132e4e" containerID="1200604f83c13c671cfc1cf0c304a3866e44a54198d025b389882c3fbd523a87" exitCode=0 Nov 03 20:15:36.266711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.253498 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" event={"ID":"e3e0b48e-8d31-489e-8759-281e75132e4e","Type":"ContainerDied","Data":"1200604f83c13c671cfc1cf0c304a3866e44a54198d025b389882c3fbd523a87"} Nov 03 20:15:36.270827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.270678 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" Nov 03 20:15:36.271284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.270677 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf" event={"ID":"4d1396f0-a7c9-4c84-b169-a359a01fc45b","Type":"ContainerDied","Data":"fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406"} Nov 03 20:15:36.271284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.271202 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fadff35c13f896840075a3745418a6e30df614882534e3e64062795f8ee28406" Nov 03 20:15:36.971151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:36.965437 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:37.445209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.445181 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:37.509922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.509887 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\" (UniqueName: \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\") pod \"e3e0b48e-8d31-489e-8759-281e75132e4e\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " Nov 03 20:15:37.510067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.509989 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\") pod \"e3e0b48e-8d31-489e-8759-281e75132e4e\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " Nov 03 20:15:37.510067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.510040 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\") pod \"e3e0b48e-8d31-489e-8759-281e75132e4e\" (UID: \"e3e0b48e-8d31-489e-8759-281e75132e4e\") " Nov 03 20:15:37.510417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.510389 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca" pod "e3e0b48e-8d31-489e-8759-281e75132e4e" (UID: "e3e0b48e-8d31-489e-8759-281e75132e4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:37.511047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.511022 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:37.512368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.512342 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca" pod "e3e0b48e-8d31-489e-8759-281e75132e4e" (UID: "e3e0b48e-8d31-489e-8759-281e75132e4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:37.512941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.512918 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr" pod "e3e0b48e-8d31-489e-8759-281e75132e4e" (UID: "e3e0b48e-8d31-489e-8759-281e75132e4e"). InnerVolumeSpecName "kube-api-access-xjdvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:37.612714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.612682 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdvr\" (UniqueName: \"kubernetes.io/projected/e3e0b48e-8d31-489e-8759-281e75132e4e-kube-api-access-xjdvr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:37.612714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.612714 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3e0b48e-8d31-489e-8759-281e75132e4e-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:37.737038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:37.737005 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.083632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.083549 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.181882 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.182866 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4d1396f0-a7c9-4c84-b169-a359a01fc45b" containerName="revoke-keys" Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.182885 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1396f0-a7c9-4c84-b169-a359a01fc45b" containerName="revoke-keys" Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.182932 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e3e0b48e-8d31-489e-8759-281e75132e4e" containerName="revoke-keys" Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.182937 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e0b48e-8d31-489e-8759-281e75132e4e" containerName="revoke-keys" Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.183061 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1396f0-a7c9-4c84-b169-a359a01fc45b" containerName="revoke-keys" Nov 03 20:15:38.184914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.183075 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e0b48e-8d31-489e-8759-281e75132e4e" containerName="revoke-keys" Nov 03 20:15:38.187901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.187877 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.208391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.208354 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:15:38.287896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.287770 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0416e8f1-openshift-default-dockercfg-lg7qh\"" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.347659 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.347768 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.347871 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.347992 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.348033 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.348077 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.348174 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.348317 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qssj\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.351042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.348433 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.413172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.409028 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453550 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453633 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453746 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453790 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453929 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.453977 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.454021 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.454123 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.456390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.454306 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qssj\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.485238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.485203 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.515282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.492434 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.515282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.493347 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.515282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.493811 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.550026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.549993 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.599136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.584478 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" event={"ID":"e3e0b48e-8d31-489e-8759-281e75132e4e","Type":"ContainerDied","Data":"b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39"} Nov 03 20:15:38.599136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.584543 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7409b72bb11f06a83f08747f3d1123963d5556d41cc0aad0e20b21b00c18c39" Nov 03 20:15:38.599136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.584614 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:15:38.611902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.601563 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:38.611902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.604781 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qssj\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.660892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.648754 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.660892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.654676 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.675661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.675532 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\") pod \"e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:38.810130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:38.806443 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:39.150931 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:39.150861 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f91fdcd_b621_49c0_805c_f29fe9056180.slice/crio-1e0031b199b9399a7551ece7619465eb8c5668f92e0b15f397258ed933c47642 WatchSource:0}: Error finding container 1e0031b199b9399a7551ece7619465eb8c5668f92e0b15f397258ed933c47642: Status 404 returned error can't find the container with id 1e0031b199b9399a7551ece7619465eb8c5668f92e0b15f397258ed933c47642 Nov 03 20:15:39.151473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.151449 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:15:39.153658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.153628 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:39.153729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.153712 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:39.153797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.153740 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:39.410779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.410698 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:39.604021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.603983 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" event={"ID":"9f91fdcd-b621-49c0-805c-f29fe9056180","Type":"ContainerStarted","Data":"1896c0435990a12db94186f1c624eed2dd4c65a4db82d76c38f01c7b16f340a5"} Nov 03 20:15:39.604559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.604039 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" event={"ID":"9f91fdcd-b621-49c0-805c-f29fe9056180","Type":"ContainerStarted","Data":"1e0031b199b9399a7551ece7619465eb8c5668f92e0b15f397258ed933c47642"} Nov 03 20:15:39.637179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.637125 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" podStartSLOduration=1.63709378 podStartE2EDuration="1.63709378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:39.634098679 +0000 UTC m=+1080.359106275" watchObservedRunningTime="2026-11-03 20:15:39.63709378 +0000 UTC m=+1080.362101375" Nov 03 20:15:39.815195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:39.807122 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:40.463095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.463064 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.813980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.813914 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:40.878010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.877960 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.941934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.941896 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:15:40.950143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.950077 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:40.972453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.972428 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:15:40.972686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:40.972667 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c847b846-openshift-default-dockercfg-8w4lp\"" Nov 03 20:15:41.163133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.144566 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.189239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.189197 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.189497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.189476 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.189636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.189622 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4t5r\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.189804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.189779 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.215410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.215370 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.215686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.215668 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.215811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.215797 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.235137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.219247 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.235652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.165239 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.339696 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.340602 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.340715 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.341547 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.341812 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342480 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342506 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342532 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4t5r\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342648 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.343135 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.342244 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.344673 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.351139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.350425 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.434184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.432213 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4t5r\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.442167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.439402 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.442167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.439902 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.464981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.460177 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\") pod \"e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.464981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.460364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:41.569551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.569489 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:41.650166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.644830 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:41.703354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.703321 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:41.721170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:41.721072 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:15:42.008056 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:42.008022 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee76231_e1cc_4ee5_83fc_91db4797d25d.slice/crio-40fae214739aa27dce16f4166cbbef0f03de1bc70e74c1858ebf7795bf5d822b WatchSource:0}: Error finding container 40fae214739aa27dce16f4166cbbef0f03de1bc70e74c1858ebf7795bf5d822b: Status 404 returned error can't find the container with id 40fae214739aa27dce16f4166cbbef0f03de1bc70e74c1858ebf7795bf5d822b Nov 03 20:15:42.008561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.008325 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:15:42.011142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.011094 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:42.011242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.011175 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:42.011242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.011223 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:42.417940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.417874 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.715309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.715222 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" event={"ID":"0ee76231-e1cc-4ee5-83fc-91db4797d25d","Type":"ContainerStarted","Data":"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c"} Nov 03 20:15:42.715309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.715289 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" event={"ID":"0ee76231-e1cc-4ee5-83fc-91db4797d25d","Type":"ContainerStarted","Data":"40fae214739aa27dce16f4166cbbef0f03de1bc70e74c1858ebf7795bf5d822b"} Nov 03 20:15:42.749172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.749099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.777336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.777261 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" podStartSLOduration=2.777248286 podStartE2EDuration="2.777248286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:42.772984482 +0000 UTC m=+1083.497992079" watchObservedRunningTime="2026-11-03 20:15:42.777248286 +0000 UTC m=+1083.502255941" Nov 03 20:15:42.970392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:42.970312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:43.571376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:43.571345 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:43.576754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:43.576730 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:43.723385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:43.723351 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:43.724229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:43.724203 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:15:44.467452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:44.467413 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:44.683696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:44.683660 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:44.960130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:44.956263 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:45.992918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:45.992885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.347182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:46.346695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.521131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:46.511686 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.658733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:46.658691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:47.301130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:47.270786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:47.509165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:47.506698 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:47.653322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:47.653288 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:48.335143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.335096 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:48.897216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.897181 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np"] Nov 03 20:15:48.904022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.903993 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np"] Nov 03 20:15:48.904214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.904093 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:48.907701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.907677 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:48.907825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:48.907737 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:49.063640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.063605 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.063838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.063812 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.063892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.063877 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjbp\" (UniqueName: \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.165797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.165714 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.165951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.165830 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.165951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.165874 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjbp\" (UniqueName: \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.166376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.166349 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.168149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.168124 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.178313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.178287 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjbp\" (UniqueName: \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\") pod \"maas-api-revoke-sub-8550ba63-895np\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.225133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.225075 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:49.355822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.355785 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:49.557507 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:15:49.557470 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec46fb5b_7dfc_4f0f_8b03_4a15197200bf.slice/crio-37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45 WatchSource:0}: Error finding container 37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45: Status 404 returned error can't find the container with id 37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45 Nov 03 20:15:49.559740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.559715 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np"] Nov 03 20:15:49.814864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:49.814784 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" event={"ID":"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf","Type":"ContainerStarted","Data":"37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45"} Nov 03 20:15:50.282275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.282229 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:50.433216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.433180 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:50.691466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.691430 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:50.825431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.825398 5954 generic.go:356] "Generic (PLEG): container finished" podID="ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" containerID="6bc5bad2b4a3c524fcfdccd14869bcff790429f4e79ff236dfdad3f6337f7cef" exitCode=0 Nov 03 20:15:50.825600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.825474 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" event={"ID":"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf","Type":"ContainerDied","Data":"6bc5bad2b4a3c524fcfdccd14869bcff790429f4e79ff236dfdad3f6337f7cef"} Nov 03 20:15:50.832149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:50.832123 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:51.342063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.342028 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:15:51.346902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.346883 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.359827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.359801 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:15:51.425318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.425288 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0416e8f1-serving-cert\"" Nov 03 20:15:51.453622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.453590 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:51.572263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.555455 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.572992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.572906 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.578910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.578877 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.580490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.580268 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlk9\" (UniqueName: \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.592173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.581705 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:51.690675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.690634 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.710138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.702279 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.710138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.702414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlk9\" (UniqueName: \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.710138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.702522 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.710810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.691745 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.717079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.717050 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.718971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.717900 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.729705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.729673 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlk9\" (UniqueName: \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\") pod \"maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:51.779340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.779312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:51.971131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:51.970464 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:52.094932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.094905 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:52.287450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.285396 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\" (UniqueName: \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\") pod \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " Nov 03 20:15:52.287450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.285480 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\") pod \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " Nov 03 20:15:52.287450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.285583 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\") pod \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\" (UID: \"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf\") " Nov 03 20:15:52.305193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.304406 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca" pod "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" (UID: "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:52.314186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.305420 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:52.318128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.316615 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca" pod "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" (UID: "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:52.413189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.412795 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:52.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.421389 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp" pod "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" (UID: "ec46fb5b-7dfc-4f0f-8b03-4a15197200bf"). InnerVolumeSpecName "kube-api-access-vpjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:52.523837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.523759 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:15:52.532205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.526407 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjbp\" (UniqueName: \"kubernetes.io/projected/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf-kube-api-access-vpjbp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:52.538605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.538576 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:15:52.671598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.671519 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:52.851530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.851489 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" event={"ID":"fd430c50-337a-4b25-8c81-7cc0ed52fe82","Type":"ContainerStarted","Data":"7c3beeaa84fddc0240f46017a025c7529b4834340a9235338a4714fa665d2568"} Nov 03 20:15:52.854788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.854756 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" event={"ID":"ec46fb5b-7dfc-4f0f-8b03-4a15197200bf","Type":"ContainerDied","Data":"37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45"} Nov 03 20:15:52.855563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.855542 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37511b7c09cfd16d4b113877f5ae030d2cf35af98ade21e859ddb78145395a45" Nov 03 20:15:52.855704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:52.855153 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np" Nov 03 20:15:53.573541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:53.573507 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:53.865044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:53.865009 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" event={"ID":"fd430c50-337a-4b25-8c81-7cc0ed52fe82","Type":"ContainerStarted","Data":"1fc5cb854cebe3dcc4e97351f8d3e95e18c3fef0b40390960fb7ac9116a69e17"} Nov 03 20:15:53.865276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:53.865256 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:15:53.888361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:53.888305 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" podStartSLOduration=2.408721834 podStartE2EDuration="2.888293939s" totalImagesPullingTime="479.572105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:53.884929492 +0000 UTC m=+1094.609937088" watchObservedRunningTime="2026-11-03 20:15:53.888293939 +0000 UTC m=+1094.613301533" Nov 03 20:15:54.409716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:54.409659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.510956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:54.510925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.617741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:54.617697 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.426689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:55.426656 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:56.117772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:56.117735 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:56.379242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:56.372878 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:56.488439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:56.488406 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:56.598735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:56.598685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:57.569704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:57.569654 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.728142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:57.727253 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.578874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.578826 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.632892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.628222 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.632892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.628757 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:15:58.758013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.747575 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:58.758013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.755383 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:15:58.988362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:58.988284 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.055135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.041653 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.141064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.089197 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.582319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.582275 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.607132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.584196 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:15:59.884189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.841608 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:59.884189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.841683 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.884189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.846187 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:59.884189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:15:59.847461 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:00.121132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.077713 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.121132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.097379 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:00.121132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.098826 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.121132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.099358 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:00.153187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.145435 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.270425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.270382 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:00.383477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.383443 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.383709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.383684 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.545188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.481223 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.545188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.488272 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:16:00.601254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.601200 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.650680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.650287 5954 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.650680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.650368 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.693147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.692935 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.693147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.692998 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.848306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.848201 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:00.970903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:00.848526 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.081398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.081344 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.312287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.162651 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:16:01.919118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.183083 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:16:02.221099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.641808 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:16:02.225214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.712482 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:02.226755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.808710 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:02.238471 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:16:02.238436 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:16:02.238693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.988399 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.239607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.988439 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:02.241143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.201075 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:16:02.242452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:01.820365 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podUID="3602a4e5-7b74-44e0-a640-3e608c97bf19" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.245405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.245367 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:02.245584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.245557 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.501902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.501770 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podUID="3602a4e5-7b74-44e0-a640-3e608c97bf19" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.716067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.715881 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.729074 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:16:02.729030 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.304s" Nov 03 20:16:02.729894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.729872 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:16:02.731074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.730915 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:16:02.731074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.731021 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:16:02.731074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.731077 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:02.731310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.731179 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:02.731861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.731833 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:16:02.732165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.731981 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:16:02.732165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.732021 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:16:02.734207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:02.734178 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:16:03.356708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:03.356678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:03.616332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:03.616246 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:03.730835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:03.730796 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:03.905839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:03.905743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:04.582509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.582477 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:04.583293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.583273 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" containerName="revoke-keys" Nov 03 20:16:04.583293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.583291 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" containerName="revoke-keys" Nov 03 20:16:04.583475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.583399 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" containerName="revoke-keys" Nov 03 20:16:04.587726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.587707 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.598467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.598447 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e44e2278-serving-cert\"" Nov 03 20:16:04.605521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.605496 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:04.659790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.659759 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:04.691522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.691492 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.691719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.691698 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.691836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.691822 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.691938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.691925 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4z9\" (UniqueName: \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.787127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.785798 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:04.834200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.833669 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4z9\" (UniqueName: \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.834200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.833850 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.834200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.834015 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.839523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.835326 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.839523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.836718 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.860936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.860892 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.870349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.869402 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.872467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.872440 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4z9\" (UniqueName: \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\") pod \"maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:04.905119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:04.905068 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:05.251016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:05.250991 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:05.793068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:05.793035 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" event={"ID":"53b42f85-bdb2-4996-926e-9b925f544e72","Type":"ContainerStarted","Data":"aec982df9bf469973c5fe872a5b71fc637f3b47add17aecc418f9b6cd7592e83"} Nov 03 20:16:05.855750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:05.855720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:06.761017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:06.760986 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:06.895613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:06.893469 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" event={"ID":"53b42f85-bdb2-4996-926e-9b925f544e72","Type":"ContainerStarted","Data":"d487014b98f9c40417d31bd6281e0d90df6af8da66c3ed588887dce3bbb573c7"} Nov 03 20:16:06.895613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:06.894798 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:06.936467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:06.936413 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" podStartSLOduration=2.404391761 podStartE2EDuration="2.936399107s" totalImagesPullingTime="532.007346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:06.919316103 +0000 UTC m=+1107.644323699" watchObservedRunningTime="2026-11-03 20:16:06.936399107 +0000 UTC m=+1107.661406702" Nov 03 20:16:07.284863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:07.284830 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:07.399071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:07.399026 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:07.508461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:07.508427 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:07.737366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:07.737327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:08.901790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:08.901755 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.139089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:10.139038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.908049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:10.908009 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:11.580384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:11.580351 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:11.728309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:11.728274 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:11.877036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:11.877000 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.019744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.019597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.165835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.165728 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.338181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.336463 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn"] Nov 03 20:16:12.346563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.346529 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn"] Nov 03 20:16:12.346732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.346642 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.352191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.352151 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:12.352191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.352180 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:16:12.482290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.482210 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnhl\" (UniqueName: \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.482509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.482487 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.482635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.482619 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.584914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.584882 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.585410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.585022 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.585410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.585179 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnhl\" (UniqueName: \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.585756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.585737 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.587397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.587374 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.599364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.599340 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnhl\" (UniqueName: \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\") pod \"maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.669409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.669380 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:12.979921 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:16:12.979893 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dcb277_86d1_459f_9521_eb3d9d16c4d2.slice/crio-1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f WatchSource:0}: Error finding container 1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f: Status 404 returned error can't find the container with id 1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f Nov 03 20:16:12.982202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:12.982176 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn"] Nov 03 20:16:13.083931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:13.083898 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" event={"ID":"42dcb277-86d1-459f-9521-eb3d9d16c4d2","Type":"ContainerStarted","Data":"1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f"} Nov 03 20:16:13.229548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:13.229501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:13.943432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:13.943408 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:14.079645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:14.079613 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:14.105632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:14.105599 5954 generic.go:356] "Generic (PLEG): container finished" podID="42dcb277-86d1-459f-9521-eb3d9d16c4d2" containerID="785b2fbc83ab27cc94a5f5a0f0bbf020d92d768939e138eb669db7873009c2fa" exitCode=0 Nov 03 20:16:14.105802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:14.105684 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" event={"ID":"42dcb277-86d1-459f-9521-eb3d9d16c4d2","Type":"ContainerDied","Data":"785b2fbc83ab27cc94a5f5a0f0bbf020d92d768939e138eb669db7873009c2fa"} Nov 03 20:16:14.901844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:14.901793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.298699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.298664 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:15.372819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.372790 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\") pod \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " Nov 03 20:16:15.373184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.373150 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\" (UniqueName: \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\") pod \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " Nov 03 20:16:15.373334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.373192 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca" pod "42dcb277-86d1-459f-9521-eb3d9d16c4d2" (UID: "42dcb277-86d1-459f-9521-eb3d9d16c4d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:15.373334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.373214 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\") pod \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\" (UID: \"42dcb277-86d1-459f-9521-eb3d9d16c4d2\") " Nov 03 20:16:15.373974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.373950 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:15.375435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.375410 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl" pod "42dcb277-86d1-459f-9521-eb3d9d16c4d2" (UID: "42dcb277-86d1-459f-9521-eb3d9d16c4d2"). InnerVolumeSpecName "kube-api-access-nfnhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:15.376338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.376311 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca" pod "42dcb277-86d1-459f-9521-eb3d9d16c4d2" (UID: "42dcb277-86d1-459f-9521-eb3d9d16c4d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:15.475039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.474978 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnhl\" (UniqueName: \"kubernetes.io/projected/42dcb277-86d1-459f-9521-eb3d9d16c4d2-kube-api-access-nfnhl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:15.475039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.475004 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dcb277-86d1-459f-9521-eb3d9d16c4d2-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:15.544384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.544353 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.647146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.647085 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.866759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:15.866722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:16.136868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:16.136785 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" Nov 03 20:16:16.136868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:16.136814 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn" event={"ID":"42dcb277-86d1-459f-9521-eb3d9d16c4d2","Type":"ContainerDied","Data":"1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f"} Nov 03 20:16:16.136868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:16.136867 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3fc2cfa42a8580ee264aeec8bdd2ab9f88ac8e6351095c5e62b5f919e6281f" Nov 03 20:16:16.870709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:16.870678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:17.687545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:17.687509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:18.399309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:18.399275 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:18.948063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:18.948030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.048271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:19.048234 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.169760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:19.168606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.551847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:19.551806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.895202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:19.895169 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.771229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:20.771190 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:21.641708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:21.641671 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.404177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:22.404146 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.506252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:22.506212 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.604803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:22.604772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:23.301659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:23.301617 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.081603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:24.081572 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.791454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:24.791421 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:25.491770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:25.491739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:25.691628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:25.691597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:25.935167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:25.928845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.033751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:26.033715 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.491270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:26.491028 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.608142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:26.606614 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.986257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:26.986229 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:16:27.257182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.244688 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:16:27.257182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.244967 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" podUID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" containerName="maas-api" containerID="cri-o://1fc5cb854cebe3dcc4e97351f8d3e95e18c3fef0b40390960fb7ac9116a69e17" gracePeriod=30 Nov 03 20:16:27.311594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.311555 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:27.610255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.590625 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:27.721867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.721674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:27.772257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.772221 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn"] Nov 03 20:16:27.784540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.784499 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0416e8f1-b9hkn"] Nov 03 20:16:27.921469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:27.921380 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:28.012559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.011788 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:16:28.012559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.012047 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" podUID="9f91fdcd-b621-49c0-805c-f29fe9056180" containerName="istio-proxy" containerID="cri-o://1896c0435990a12db94186f1c624eed2dd4c65a4db82d76c38f01c7b16f340a5" gracePeriod=30 Nov 03 20:16:28.121023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.119482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:28.464681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.464637 5954 generic.go:356] "Generic (PLEG): container finished" podID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" containerID="1fc5cb854cebe3dcc4e97351f8d3e95e18c3fef0b40390960fb7ac9116a69e17" exitCode=0 Nov 03 20:16:28.469154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.467481 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" event={"ID":"fd430c50-337a-4b25-8c81-7cc0ed52fe82","Type":"ContainerDied","Data":"1fc5cb854cebe3dcc4e97351f8d3e95e18c3fef0b40390960fb7ac9116a69e17"} Nov 03 20:16:28.734215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:28.713139 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:29.610128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:29.604194 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:29.771165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:29.767084 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.163213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.135501 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dcb277-86d1-459f-9521-eb3d9d16c4d2" path="/var/lib/kubelet/pods/42dcb277-86d1-459f-9521-eb3d9d16c4d2/volumes" Nov 03 20:16:30.163213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.160013 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:16:30.192131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.183913 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:30.239925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.234133 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" event={"ID":"fd430c50-337a-4b25-8c81-7cc0ed52fe82","Type":"ContainerDied","Data":"7c3beeaa84fddc0240f46017a025c7529b4834340a9235338a4714fa665d2568"} Nov 03 20:16:30.239925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.234227 5954 scope.go:122] "RemoveContainer" containerID="1fc5cb854cebe3dcc4e97351f8d3e95e18c3fef0b40390960fb7ac9116a69e17" Nov 03 20:16:30.239925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.234437 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb" Nov 03 20:16:30.349242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.349206 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\" (UniqueName: \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\") pod \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " Nov 03 20:16:30.378288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.369547 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:30.378874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.378841 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\") pod \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " Nov 03 20:16:30.385405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.385369 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\") pod \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " Nov 03 20:16:30.429199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.394331 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca" pod "fd430c50-337a-4b25-8c81-7cc0ed52fe82" (UID: "fd430c50-337a-4b25-8c81-7cc0ed52fe82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.429523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.429485 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\") pod \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\" (UID: \"fd430c50-337a-4b25-8c81-7cc0ed52fe82\") " Nov 03 20:16:30.448405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.448357 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9" pod "fd430c50-337a-4b25-8c81-7cc0ed52fe82" (UID: "fd430c50-337a-4b25-8c81-7cc0ed52fe82"). InnerVolumeSpecName "kube-api-access-jvlk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:30.471808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.471756 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls" pod "fd430c50-337a-4b25-8c81-7cc0ed52fe82" (UID: "fd430c50-337a-4b25-8c81-7cc0ed52fe82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:30.496281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.488214 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.584751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.564221 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd430c50-337a-4b25-8c81-7cc0ed52fe82-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.584751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.564273 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlk9\" (UniqueName: \"kubernetes.io/projected/fd430c50-337a-4b25-8c81-7cc0ed52fe82-kube-api-access-jvlk9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.677127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.669243 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs" pod "fd430c50-337a-4b25-8c81-7cc0ed52fe82" (UID: "fd430c50-337a-4b25-8c81-7cc0ed52fe82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:30.677127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.675242 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd430c50-337a-4b25-8c81-7cc0ed52fe82-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.783749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:30.783658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:30.811251 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:16:30.806313 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:16:31.139133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.126038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:31.139458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.135004 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.135042 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.135064 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.135096 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.142300 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.183231 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.186415 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.263194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.261474 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.727552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.727514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:31.910135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:31.895618 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:32.155131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.141435 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.155131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.141990 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:16:32.219756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.156438 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.245155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.159530 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:16:32.281893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.280325 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:16:32.281893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.186380 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.281893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.208549 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:16:32.317277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.317246 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:16:32.317608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.317590 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:32.317798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.317775 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:16:32.343913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.343877 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:32.345068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.159606 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.399718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.399661 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.409182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.400729 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:16:32.562141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.426897 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:16:32.562141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.509948 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:32.899809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:32.899766 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded" Nov 03 20:16:33.113265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.113225 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0416e8f1-6cc466cc74-5hctb"] Nov 03 20:16:33.178289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.149142 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:33.214374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.213963 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:33.229955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.229500 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:33.275150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.257875 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:16:33.362371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.362308 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:16:33.381449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.381408 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:33.405133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.392785 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.426173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.402238 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:16:33.888337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:33.887750 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.071563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.071505 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" path="/var/lib/kubelet/pods/fd430c50-337a-4b25-8c81-7cc0ed52fe82/volumes" Nov 03 20:16:34.204185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.200465 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:34.285247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.285203 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.397131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.369815 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:34.452165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.420368 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.453569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.436263 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:16:34.505028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.472968 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.671160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.548381 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.703150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.474829 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:34.754362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.754307 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.950137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.912613 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:16:34.950137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:34.913918 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerName="istio-proxy" containerID="cri-o://aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c" gracePeriod=30 Nov 03 20:16:35.139155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.139088 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:16:35.141565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.141474 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:35.219191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.219023 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.517048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.516952 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:16:35.532926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.532873 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:16:35.661616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.661551 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.744247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:35.744206 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.168526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.168475 5954 generic.go:356] "Generic (PLEG): container finished" podID="9f91fdcd-b621-49c0-805c-f29fe9056180" containerID="1896c0435990a12db94186f1c624eed2dd4c65a4db82d76c38f01c7b16f340a5" exitCode=0 Nov 03 20:16:36.182905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.182861 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:36.257722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.257679 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.380480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.380416 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" event={"ID":"9f91fdcd-b621-49c0-805c-f29fe9056180","Type":"ContainerDied","Data":"1896c0435990a12db94186f1c624eed2dd4c65a4db82d76c38f01c7b16f340a5"} Nov 03 20:16:36.535188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.529410 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.626977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.626933 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.768136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.760554 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:36.768136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.764211 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.809151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.809098 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:16:36.815787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:36.815760 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:37.096502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.096407 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:37.265262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.223734 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.269132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.265882 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:37.275551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.275520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:37.287244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.287205 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:37.357284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.346433 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podUID="3602a4e5-7b74-44e0-a640-3e608c97bf19" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.510128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.497991 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.659174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.509440 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:37.659174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.549700 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-29vkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:37.659174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.650314 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.659174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.634524 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.855141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.799930 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.919705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:37.822594 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.290556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.290460 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.328258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.317706 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.338240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.338206 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:16:38.364534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.350607 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:38.405131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.371192 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:16:38.505211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.505165 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.506312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.506286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:38.630460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.524802 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.715339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.687446 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 03 20:16:38.715339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.687701 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:16:38.715339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.688299 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:38.717924 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.529557 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gl4rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:38.718240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.718209 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.730838 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.736594 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.743730 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.743913 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.744035 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.744194 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.744260 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.744317 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.764165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.744361 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\") pod \"9f91fdcd-b621-49c0-805c-f29fe9056180\" (UID: \"9f91fdcd-b621-49c0-805c-f29fe9056180\") " Nov 03 20:16:38.765412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.765362 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:38.797208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.792129 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:38.797863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.797815 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:38.799284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.799260 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:38.834571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.834510 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.863061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.856056 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.865272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.865227 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:38.865714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.865678 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:38.917344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.917240 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.930330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.930288 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.930572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.926699 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:38.999817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.992415 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:38.999817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.992454 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:38.999817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.992471 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f91fdcd-b621-49c0-805c-f29fe9056180-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:38.999817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:38.938799 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:39.103360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.103310 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:16:39.105225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.103737 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "kube-api-access-8qssj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:39.156987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.156949 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" event={"ID":"9f91fdcd-b621-49c0-805c-f29fe9056180","Type":"ContainerDied","Data":"1e0031b199b9399a7551ece7619465eb8c5668f92e0b15f397258ed933c47642"} Nov 03 20:16:39.179456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.179382 5954 scope.go:122] "RemoveContainer" containerID="1896c0435990a12db94186f1c624eed2dd4c65a4db82d76c38f01c7b16f340a5" Nov 03 20:16:39.184545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.184517 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr" Nov 03 20:16:39.246214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.244223 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:39.246214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.244593 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:39.246214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.244618 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qssj\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-kube-api-access-8qssj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:39.246214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.244633 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:39.280686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.280640 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token" pod "9f91fdcd-b621-49c0-805c-f29fe9056180" (UID: "9f91fdcd-b621-49c0-805c-f29fe9056180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:39.301709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.299068 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.320248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.320204 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.389706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.389670 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f91fdcd-b621-49c0-805c-f29fe9056180-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:39.600386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.600347 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:16:39.600666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.600641 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0416e8f1-openshift-default-697859876c-5pcjr"] Nov 03 20:16:39.676718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.676686 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:16:39.688658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.688610 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.716949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.715564 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:16:39.758143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.747783 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:16:39.792162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.787012 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:16:39.859360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.858351 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:16:39.877218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.861694 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:16:39.877218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.874184 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:39.877218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.876265 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:16:39.883506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:39.883485 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:16:40.005512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:40.005400 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:40.796031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:40.796003 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:41.406860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:41.406828 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f91fdcd-b621-49c0-805c-f29fe9056180" path="/var/lib/kubelet/pods/9f91fdcd-b621-49c0-805c-f29fe9056180/volumes" Nov 03 20:16:42.085592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:42.085557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:42.677782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:42.677750 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:42.678234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:42.678203 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" podUID="53b42f85-bdb2-4996-926e-9b925f544e72" containerName="maas-api" containerID="cri-o://d487014b98f9c40417d31bd6281e0d90df6af8da66c3ed588887dce3bbb573c7" gracePeriod=30 Nov 03 20:16:42.742400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:42.742367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.049279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.042633 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.138736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.138695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.327283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.327201 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.362115 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.362073 5954 generic.go:356] "Generic (PLEG): container finished" podID="53b42f85-bdb2-4996-926e-9b925f544e72" containerID="d487014b98f9c40417d31bd6281e0d90df6af8da66c3ed588887dce3bbb573c7" exitCode=0 Nov 03 20:16:43.362266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.362219 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" event={"ID":"53b42f85-bdb2-4996-926e-9b925f544e72","Type":"ContainerDied","Data":"d487014b98f9c40417d31bd6281e0d90df6af8da66c3ed588887dce3bbb573c7"} Nov 03 20:16:43.362266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.362257 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" event={"ID":"53b42f85-bdb2-4996-926e-9b925f544e72","Type":"ContainerDied","Data":"aec982df9bf469973c5fe872a5b71fc637f3b47add17aecc418f9b6cd7592e83"} Nov 03 20:16:43.362373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.362275 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec982df9bf469973c5fe872a5b71fc637f3b47add17aecc418f9b6cd7592e83" Nov 03 20:16:43.368848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.368824 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:43.463070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.463003 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.539381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.539333 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\") pod \"53b42f85-bdb2-4996-926e-9b925f544e72\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " Nov 03 20:16:43.539752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.539724 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\") pod \"53b42f85-bdb2-4996-926e-9b925f544e72\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " Nov 03 20:16:43.539944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.539920 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\") pod \"53b42f85-bdb2-4996-926e-9b925f544e72\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " Nov 03 20:16:43.540119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.540084 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\" (UniqueName: \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\") pod \"53b42f85-bdb2-4996-926e-9b925f544e72\" (UID: \"53b42f85-bdb2-4996-926e-9b925f544e72\") " Nov 03 20:16:43.570138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.569841 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9" pod "53b42f85-bdb2-4996-926e-9b925f544e72" (UID: "53b42f85-bdb2-4996-926e-9b925f544e72"). InnerVolumeSpecName "kube-api-access-qp4z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:43.570245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.570198 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca" pod "53b42f85-bdb2-4996-926e-9b925f544e72" (UID: "53b42f85-bdb2-4996-926e-9b925f544e72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:43.649540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.648915 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b42f85-bdb2-4996-926e-9b925f544e72-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:43.649540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.648945 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4z9\" (UniqueName: \"kubernetes.io/projected/53b42f85-bdb2-4996-926e-9b925f544e72-kube-api-access-qp4z9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:43.650000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.649951 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs" pod "53b42f85-bdb2-4996-926e-9b925f544e72" (UID: "53b42f85-bdb2-4996-926e-9b925f544e72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:43.651366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.651326 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls" pod "53b42f85-bdb2-4996-926e-9b925f544e72" (UID: "53b42f85-bdb2-4996-926e-9b925f544e72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:43.817165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.815946 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:43.817165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:43.815990 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b42f85-bdb2-4996-926e-9b925f544e72-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:44.337810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.337765 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.371804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.371768 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx" Nov 03 20:16:44.401998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.401965 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:44.410783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.410750 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e44e2278-5dc898849f-thdfx"] Nov 03 20:16:44.440130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.440083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.682507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.682468 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.788290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.788239 5954 patch_prober.go:29] interesting pod/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:44.788523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:44.788319 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:45.407897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:45.407863 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b42f85-bdb2-4996-926e-9b925f544e72" path="/var/lib/kubelet/pods/53b42f85-bdb2-4996-926e-9b925f544e72/volumes" Nov 03 20:16:45.672546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:45.672460 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:46.497504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.497469 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:46.938087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.938062 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:16:46.998220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998188 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998254 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998330 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998365 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998406 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998452 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998545 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998527 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.998654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998589 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.998654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998623 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998659 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\") pod \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\" (UID: \"0ee76231-e1cc-4ee5-83fc-91db4797d25d\") " Nov 03 20:16:46.998861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998754 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.999023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.998991 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:46.999201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999172 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.999593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999573 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.999678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999597 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.999678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999612 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.999678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999624 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.999678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:46.999636 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.001265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.001239 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:47.001366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.001298 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "kube-api-access-q4t5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:47.001366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.001325 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:47.001444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.001363 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo" pod "0ee76231-e1cc-4ee5-83fc-91db4797d25d" (UID: "0ee76231-e1cc-4ee5-83fc-91db4797d25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:16:47.100843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.100754 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4t5r\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-kube-api-access-q4t5r\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.100843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.100789 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.100843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.100802 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.100843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.100815 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee76231-e1cc-4ee5-83fc-91db4797d25d-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.338937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.338903 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:16:47.339893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.339867 5954 cpu_manager.go:414] "containerMap: removing container" podUID="42dcb277-86d1-459f-9521-eb3d9d16c4d2" containerName="revoke-keys" Nov 03 20:16:47.340092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340068 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dcb277-86d1-459f-9521-eb3d9d16c4d2" containerName="revoke-keys" Nov 03 20:16:47.340228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340163 5954 cpu_manager.go:414] "containerMap: removing container" podUID="53b42f85-bdb2-4996-926e-9b925f544e72" containerName="maas-api" Nov 03 20:16:47.340228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340175 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b42f85-bdb2-4996-926e-9b925f544e72" containerName="maas-api" Nov 03 20:16:47.340228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340195 5954 cpu_manager.go:414] "containerMap: removing container" podUID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" containerName="maas-api" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340202 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" containerName="maas-api" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340255 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerName="istio-proxy" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340265 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerName="istio-proxy" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340306 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9f91fdcd-b621-49c0-805c-f29fe9056180" containerName="istio-proxy" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340315 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f91fdcd-b621-49c0-805c-f29fe9056180" containerName="istio-proxy" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340508 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f91fdcd-b621-49c0-805c-f29fe9056180" containerName="istio-proxy" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340522 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dcb277-86d1-459f-9521-eb3d9d16c4d2" containerName="revoke-keys" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340538 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b42f85-bdb2-4996-926e-9b925f544e72" containerName="maas-api" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340547 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd430c50-337a-4b25-8c81-7cc0ed52fe82" containerName="maas-api" Nov 03 20:16:47.340793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.340557 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerName="istio-proxy" Nov 03 20:16:47.343910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.343890 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.352040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.351912 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ffd1cf5a-serving-cert\"" Nov 03 20:16:47.352397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.352376 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:16:47.377549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.377520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.402729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.402700 5954 generic.go:356] "Generic (PLEG): container finished" podID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" containerID="aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c" exitCode=0 Nov 03 20:16:47.402899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.402800 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" Nov 03 20:16:47.405079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.405056 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.405253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.405237 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.405321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.405307 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.405485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.405463 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pslvt\" (UniqueName: \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.409720 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.409694 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" event={"ID":"0ee76231-e1cc-4ee5-83fc-91db4797d25d","Type":"ContainerDied","Data":"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c"} Nov 03 20:16:47.409826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.409746 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59" event={"ID":"0ee76231-e1cc-4ee5-83fc-91db4797d25d","Type":"ContainerDied","Data":"40fae214739aa27dce16f4166cbbef0f03de1bc70e74c1858ebf7795bf5d822b"} Nov 03 20:16:47.409826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.409769 5954 scope.go:122] "RemoveContainer" containerID="aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c" Nov 03 20:16:47.422461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.422443 5954 scope.go:122] "RemoveContainer" containerID="aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c" Nov 03 20:16:47.422836 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:16:47.422809 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c\": container with ID starting with aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c not found: ID does not exist" containerID="aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c" Nov 03 20:16:47.422935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.422846 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c"} err="failed to get container status \"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c\": rpc error: code = NotFound desc = could not find container \"aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c\": container with ID starting with aa5ca48ea7d510167ca149e1031e2365c6bc4b78df8cc3e462575f6b4accf76c not found: ID does not exist" Nov 03 20:16:47.434304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.434283 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:16:47.443653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.443633 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c847b846-openshift-default-5d8f866d9c-t6w59"] Nov 03 20:16:47.472659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.472617 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.506454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.506425 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.506900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.506519 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.506900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.506628 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pslvt\" (UniqueName: \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.506900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.506769 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.507523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.507496 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.508937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.508915 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.509291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.509274 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.518698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.518674 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pslvt\" (UniqueName: \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\") pod \"maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.600129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.599576 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.674172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.673399 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:47.729914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:47.729862 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:48.146985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:48.146953 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:16:48.291988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:48.291946 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:48.445780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:48.445660 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" event={"ID":"3128325d-2655-44df-a463-2b6036ce252e","Type":"ContainerStarted","Data":"08cab568b4fbd45cfbe3d3e753c0f26c25d919b3bf0e0072d8ff30bce133761c"} Nov 03 20:16:49.042576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.042544 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:16:49.046445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.046424 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.062620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.062379 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-11b0d340-serving-cert\"" Nov 03 20:16:49.066999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.066971 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:16:49.089519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.089398 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.089680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.089584 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.089790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.089760 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.089876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.089859 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzt8\" (UniqueName: \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.095377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.095344 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:49.201085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.201042 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.202408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.202378 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.202776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.202728 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.203511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.203486 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzt8\" (UniqueName: \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.207280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.207255 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.211818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.211012 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.219907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.219739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:49.227760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.227694 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.261391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.261347 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzt8\" (UniqueName: \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\") pod \"maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.341184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.324931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:49.370811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.370766 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:49.409428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.409389 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee76231-e1cc-4ee5-83fc-91db4797d25d" path="/var/lib/kubelet/pods/0ee76231-e1cc-4ee5-83fc-91db4797d25d/volumes" Nov 03 20:16:49.465968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.465350 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" event={"ID":"3128325d-2655-44df-a463-2b6036ce252e","Type":"ContainerStarted","Data":"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7"} Nov 03 20:16:49.469454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.466475 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:49.489659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.489601 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" podStartSLOduration=1.7983901549999999 podStartE2EDuration="2.489581644s" totalImagesPullingTime="691.191489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:49.488040392 +0000 UTC m=+1150.213047990" watchObservedRunningTime="2026-11-03 20:16:49.489581644 +0000 UTC m=+1150.214589242" Nov 03 20:16:49.739981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.739949 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:16:49.980925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:49.980886 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:50.587326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.586126 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" event={"ID":"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c","Type":"ContainerStarted","Data":"2e8b0791c49a5d5128d4baaefb3fea967c962e9e8d7efaf05592edfefd2e3343"} Nov 03 20:16:50.587938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.587913 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:16:50.657031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.657002 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:50.657420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.657396 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" event={"ID":"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c","Type":"ContainerStarted","Data":"42c6d8349199756594f3343cdfe5d1105ee21c8ab946b32b6ebbdd5253c6d705"} Nov 03 20:16:50.657551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.657340 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.668992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.668959 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:16:50.693038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.691886 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" podStartSLOduration=1.243098016 podStartE2EDuration="1.691865908s" totalImagesPullingTime="448.767892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:50.624511853 +0000 UTC m=+1151.349519449" watchObservedRunningTime="2026-11-03 20:16:50.691865908 +0000 UTC m=+1151.416873504" Nov 03 20:16:50.720777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.720738 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.720974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.720881 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.720974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.720939 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.721133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.721009 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz96r\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.721259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.721241 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.721331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.721299 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.721413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.721388 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.795781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.795747 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:50.822836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.822792 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.822887 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.822920 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.822976 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz96r\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.823020 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.823063 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.823255 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.823921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.823897 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.825995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.825968 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.826101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.826032 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.826101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.826087 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.826237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.826224 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.860196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.836911 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:50.860196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:50.854651 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz96r\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lth6n\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:51.000623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.000589 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:51.290397 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:16:51.290335 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78769a77_eb1b_4e93_b3ff_454954858b61.slice/crio-9e0573e616767dfba2e7836d2082536a795bc055bea64f2a8a0b46598754003f WatchSource:0}: Error finding container 9e0573e616767dfba2e7836d2082536a795bc055bea64f2a8a0b46598754003f: Status 404 returned error can't find the container with id 9e0573e616767dfba2e7836d2082536a795bc055bea64f2a8a0b46598754003f Nov 03 20:16:51.292740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.292704 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:16:51.293889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.293845 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:51.294025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.293933 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:51.477092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.477051 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:51.615528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.615494 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" event={"ID":"78769a77-eb1b-4e93-b3ff-454954858b61","Type":"ContainerStarted","Data":"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3"} Nov 03 20:16:51.617125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.617077 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" event={"ID":"78769a77-eb1b-4e93-b3ff-454954858b61","Type":"ContainerStarted","Data":"9e0573e616767dfba2e7836d2082536a795bc055bea64f2a8a0b46598754003f"} Nov 03 20:16:51.617409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.617386 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:51.693752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:51.689410 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" podStartSLOduration=1.689391646 podStartE2EDuration="1.689391646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:51.676002208 +0000 UTC m=+1152.401009803" watchObservedRunningTime="2026-11-03 20:16:51.689391646 +0000 UTC m=+1152.414399241" Nov 03 20:16:52.287670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:52.287629 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:52.634993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:52.634960 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:16:53.002182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:53.001428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:53.741208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:53.741177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:54.487841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:54.487796 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:55.174423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:55.174388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:55.871277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:55.871248 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:56.558815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:56.558783 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:56.670140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:56.668196 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:16:57.207099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:57.207073 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:57.639318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:57.639289 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:16:57.993295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:57.993262 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:58.779896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:58.779864 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:59.480089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:16:59.479928 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:00.241525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:00.241484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:00.259901 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:17:00.259862 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:17:00.710129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:00.701160 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.092317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:01.092219 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.912020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:01.911983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:02.942139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:02.941214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:03.756968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:03.756920 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:04.281043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:04.281008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:04.726444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:04.726401 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-khb7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:17:04.726765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:04.726731 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:04.808248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:04.808207 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:17:04.808454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:04.808287 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:05.096776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:05.096676 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.097353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:06.097319 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.944831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:06.944794 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.312387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:07.312265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.687762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:07.687696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:08.550907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:08.550859 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:09.478563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:09.478525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:10.173075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:10.173041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:10.877559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:10.877522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:11.028129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:11.026714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:11.672123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:11.672048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:12.410341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:12.410309 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:13.040051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.040025 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b"] Nov 03 20:17:13.048607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.048580 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8df55c8-g9g6b"] Nov 03 20:17:13.057191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.057162 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf"] Nov 03 20:17:13.065483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.065457 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3c58ab-vnkpf"] Nov 03 20:17:13.106848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.106813 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:13.408926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.408884 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0382582e-778d-407b-a032-4e5cab1841f4" path="/var/lib/kubelet/pods/0382582e-778d-407b-a032-4e5cab1841f4/volumes" Nov 03 20:17:13.409591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.409569 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7607488b-e33c-4113-a22d-d4c8426a827d" path="/var/lib/kubelet/pods/7607488b-e33c-4113-a22d-d4c8426a827d/volumes" Nov 03 20:17:13.874206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:13.874168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:14.043040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:14.043006 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj"] Nov 03 20:17:14.047705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:14.047688 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:17:14.052045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:14.052019 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c465c9-tp8bj"] Nov 03 20:17:14.730979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:14.730949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:15.411150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:15.411097 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab2a3ea-22fd-49a3-9e03-03b95075a272" path="/var/lib/kubelet/pods/bab2a3ea-22fd-49a3-9e03-03b95075a272/volumes" Nov 03 20:17:15.648857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:15.648820 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:16.494120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:16.494086 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:17.205595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:17.205564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:17.989919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:17.989885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:18.391559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:18.391525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:18.496419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:18.496350 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:19.023657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:19.023622 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:19.702325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:19.702291 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:20.588439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:20.588409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:21.232675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.232631 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:17:21.232931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.232889 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" podUID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" containerName="maas-api" containerID="cri-o://2e8b0791c49a5d5128d4baaefb3fea967c962e9e8d7efaf05592edfefd2e3343" gracePeriod=30 Nov 03 20:17:21.284069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.284039 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:21.444619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.444333 5954 generic.go:356] "Generic (PLEG): container finished" podID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" containerID="2e8b0791c49a5d5128d4baaefb3fea967c962e9e8d7efaf05592edfefd2e3343" exitCode=0 Nov 03 20:17:21.444619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.444417 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" event={"ID":"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c","Type":"ContainerDied","Data":"2e8b0791c49a5d5128d4baaefb3fea967c962e9e8d7efaf05592edfefd2e3343"} Nov 03 20:17:21.574875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.574840 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:21.613140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.611374 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:17:21.702131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.701755 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\") pod \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " Nov 03 20:17:21.702131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.701831 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\" (UniqueName: \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\") pod \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " Nov 03 20:17:21.702131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.701971 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\") pod \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " Nov 03 20:17:21.702131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.702049 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\") pod \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\" (UID: \"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c\") " Nov 03 20:17:21.716132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.706317 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca" pod "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" (UID: "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:21.722127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.721628 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs" pod "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" (UID: "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:21.730355 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.730284 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8" pod "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" (UID: "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c"). InnerVolumeSpecName "kube-api-access-wkzt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:21.742430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.742232 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls" pod "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" (UID: "cf4d2e8e-83a3-431c-80e1-8e90b67cb59c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:21.830428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.801957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:21.894340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.850386 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:21.894340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.850419 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzt8\" (UniqueName: \"kubernetes.io/projected/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-kube-api-access-wkzt8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:21.894340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.850435 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:21.894340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:21.850448 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:22.009605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.009526 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.032003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.031970 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5"] Nov 03 20:17:22.039634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.039594 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221c3325-62mp5"] Nov 03 20:17:22.124133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.122732 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.479123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.469074 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" event={"ID":"cf4d2e8e-83a3-431c-80e1-8e90b67cb59c","Type":"ContainerDied","Data":"42c6d8349199756594f3343cdfe5d1105ee21c8ab946b32b6ebbdd5253c6d705"} Nov 03 20:17:22.479123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.469198 5954 scope.go:122] "RemoveContainer" containerID="2e8b0791c49a5d5128d4baaefb3fea967c962e9e8d7efaf05592edfefd2e3343" Nov 03 20:17:22.479123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.469383 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4" Nov 03 20:17:22.505275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.505250 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:17:22.510093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.510068 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-11b0d340-665b974f7c-sxgt4"] Nov 03 20:17:22.779782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.779703 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.899310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:22.899273 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.283876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:23.283842 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.603179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:23.601270 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3" path="/var/lib/kubelet/pods/8cae5a1b-6fe5-4e3f-ae87-7d896df88eb3/volumes" Nov 03 20:17:23.603179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:23.602013 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" path="/var/lib/kubelet/pods/cf4d2e8e-83a3-431c-80e1-8e90b67cb59c/volumes" Nov 03 20:17:23.825613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:23.825582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.925577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:23.925487 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:24.692775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:24.692715 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:25.393599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:25.393564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.024566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:26.024529 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f"] Nov 03 20:17:26.029377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:26.029343 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09551f22-g5d7f"] Nov 03 20:17:26.330262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:26.330182 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.865469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:26.865434 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.969062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:26.969027 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:27.183912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:27.183824 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:27.407776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:27.407734 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01260483-d289-488b-a38b-6b2c9166f290" path="/var/lib/kubelet/pods/01260483-d289-488b-a38b-6b2c9166f290/volumes" Nov 03 20:17:28.043631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:28.043581 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n"] Nov 03 20:17:28.051576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:28.051542 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d060ba-v845n"] Nov 03 20:17:28.066407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:28.066378 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:28.168787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:28.168745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:29.038705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:29.038673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:29.407491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:29.407458 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e47058-56b5-4088-872d-3c4166ee19b8" path="/var/lib/kubelet/pods/71e47058-56b5-4088-872d-3c4166ee19b8/volumes" Nov 03 20:17:29.718564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:29.718471 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.259753 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:17:30.259714 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:17:30.494703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:30.494668 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.637592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:30.632137 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.761129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:30.743946 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.293767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:31.293731 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:32.080935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:32.080900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:32.772236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:32.772200 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:34.803851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:34.803817 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.179588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:35.179554 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.712150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:35.712116 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:17:35.712707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:35.712671 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" podUID="3128325d-2655-44df-a463-2b6036ce252e" containerName="maas-api" containerID="cri-o://157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7" gracePeriod=30 Nov 03 20:17:35.751764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:35.751729 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.945077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:35.945044 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:36.029897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.029865 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:17:36.129096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.119897 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\") pod \"3128325d-2655-44df-a463-2b6036ce252e\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " Nov 03 20:17:36.129096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.120028 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\") pod \"3128325d-2655-44df-a463-2b6036ce252e\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " Nov 03 20:17:36.129096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.120098 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\") pod \"3128325d-2655-44df-a463-2b6036ce252e\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " Nov 03 20:17:36.129096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.120212 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\" (UniqueName: \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\") pod \"3128325d-2655-44df-a463-2b6036ce252e\" (UID: \"3128325d-2655-44df-a463-2b6036ce252e\") " Nov 03 20:17:36.129096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.124906 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:36.154337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.149543 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt" pod "3128325d-2655-44df-a463-2b6036ce252e" (UID: "3128325d-2655-44df-a463-2b6036ce252e"). InnerVolumeSpecName "kube-api-access-pslvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:36.154337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.149987 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca" pod "3128325d-2655-44df-a463-2b6036ce252e" (UID: "3128325d-2655-44df-a463-2b6036ce252e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:36.154337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.152133 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs" pod "3128325d-2655-44df-a463-2b6036ce252e" (UID: "3128325d-2655-44df-a463-2b6036ce252e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:36.193773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.185456 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls" pod "3128325d-2655-44df-a463-2b6036ce252e" (UID: "3128325d-2655-44df-a463-2b6036ce252e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:36.307978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.307868 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:36.427930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.423718 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.427930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.423880 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pslvt\" (UniqueName: \"kubernetes.io/projected/3128325d-2655-44df-a463-2b6036ce252e-kube-api-access-pslvt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.427930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.423905 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3128325d-2655-44df-a463-2b6036ce252e-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.427930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.423921 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128325d-2655-44df-a463-2b6036ce252e-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:36.823242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.823201 5954 generic.go:356] "Generic (PLEG): container finished" podID="3128325d-2655-44df-a463-2b6036ce252e" containerID="157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7" exitCode=0 Nov 03 20:17:36.823437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.823256 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" Nov 03 20:17:36.823437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.823287 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" event={"ID":"3128325d-2655-44df-a463-2b6036ce252e","Type":"ContainerDied","Data":"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7"} Nov 03 20:17:36.823437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.823347 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj" event={"ID":"3128325d-2655-44df-a463-2b6036ce252e","Type":"ContainerDied","Data":"08cab568b4fbd45cfbe3d3e753c0f26c25d919b3bf0e0072d8ff30bce133761c"} Nov 03 20:17:36.823437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.823369 5954 scope.go:122] "RemoveContainer" containerID="157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7" Nov 03 20:17:36.835624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.835604 5954 scope.go:122] "RemoveContainer" containerID="157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7" Nov 03 20:17:36.835918 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:17:36.835892 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7\": container with ID starting with 157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7 not found: ID does not exist" containerID="157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7" Nov 03 20:17:36.836035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.835922 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7"} err="failed to get container status \"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7\": rpc error: code = NotFound desc = could not find container \"157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7\": container with ID starting with 157f25539bad06f2fc7ab2d4dca1ea32300dca85cd3ca46a753c4d37086b94b7 not found: ID does not exist" Nov 03 20:17:36.852633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.852607 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:17:36.857064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:36.857032 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ffd1cf5a-7dfb44f6d9-5htfj"] Nov 03 20:17:37.029756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.029676 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk"] Nov 03 20:17:37.034667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.034642 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c42a4a4-wrmvk"] Nov 03 20:17:37.287503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.287427 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:37.437136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.436310 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3128325d-2655-44df-a463-2b6036ce252e" path="/var/lib/kubelet/pods/3128325d-2655-44df-a463-2b6036ce252e/volumes" Nov 03 20:17:37.445735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.445695 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53ac0e3-b9b8-4f68-a885-6fe93084a936" path="/var/lib/kubelet/pods/c53ac0e3-b9b8-4f68-a885-6fe93084a936/volumes" Nov 03 20:17:37.600037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:37.599951 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.043130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:38.042655 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf"] Nov 03 20:17:38.060237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:38.058076 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07af413-nxlpf"] Nov 03 20:17:38.266837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:38.266806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.410618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:38.410569 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.899646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:38.899606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:39.011063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:39.011030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:39.248179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:39.248054 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:39.466068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:39.463942 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2feed9-d5a0-404a-9291-230f0985aeb9" path="/var/lib/kubelet/pods/9a2feed9-d5a0-404a-9291-230f0985aeb9/volumes" Nov 03 20:17:39.815997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:39.815964 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:40.589398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:40.589366 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:41.024613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:41.024574 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc"] Nov 03 20:17:41.028966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:41.028940 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1aa1299-lghvc"] Nov 03 20:17:41.367893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:41.367859 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:41.408553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:41.408522 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275a3b30-a5c1-46b9-8cbd-9c7149e87878" path="/var/lib/kubelet/pods/275a3b30-a5c1-46b9-8cbd-9c7149e87878/volumes" Nov 03 20:17:42.114727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:42.114699 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.101288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:43.101255 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.226067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:43.226022 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.327154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:43.327045 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:44.260756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:44.260726 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:44.455050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:44.455000 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:44.568141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:44.568048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:45.241051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:45.241016 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:45.991820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:45.991782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:46.784329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:46.784287 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:47.571498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:47.571464 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:47.775519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:47.775472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:48.373551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:48.373517 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:49.016062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:49.016020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:50.045851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.045320 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:50.134671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.134630 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p"] Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.150470 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf606668-69s5p"] Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.154847 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd"] Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156081 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3128325d-2655-44df-a463-2b6036ce252e" containerName="maas-api" Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156150 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3128325d-2655-44df-a463-2b6036ce252e" containerName="maas-api" Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156242 5954 cpu_manager.go:414] "containerMap: removing container" podUID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" containerName="maas-api" Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156253 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" containerName="maas-api" Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156453 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4d2e8e-83a3-431c-80e1-8e90b67cb59c" containerName="maas-api" Nov 03 20:17:50.168370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.156475 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3128325d-2655-44df-a463-2b6036ce252e" containerName="maas-api" Nov 03 20:17:50.176006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.175913 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:50.178749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.178724 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.184291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.183353 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:17:50.184291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.183415 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:50.244186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.205530 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.252139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.244641 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzrq\" (UniqueName: \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.252833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.252796 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.349752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.349722 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd"] Nov 03 20:17:50.373242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.361008 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.373242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.361179 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzrq\" (UniqueName: \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.373242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.361419 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.375622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.375532 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.405584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.405499 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.433177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.430793 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzrq\" (UniqueName: \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\") pod \"maas-api-revoke-sub-d320a902-7vctd\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:50.626450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:50.626410 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:51.246046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:51.245962 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:51.386571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:51.383955 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd"] Nov 03 20:17:51.446911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:51.446809 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75711e96-f618-48d7-b7ad-db6f25b4f4f7" path="/var/lib/kubelet/pods/75711e96-f618-48d7-b7ad-db6f25b4f4f7/volumes" Nov 03 20:17:51.991788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:51.991757 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:52.158964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.158929 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:52.295335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.294330 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" event={"ID":"0f6c1b34-e2be-47ef-b82b-abdacb12de8f","Type":"ContainerStarted","Data":"d4d4721323e00ac769e6a6e03df1294128ab589799ad8765fec0a0705dd8675a"} Nov 03 20:17:52.295969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.295939 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" event={"ID":"0f6c1b34-e2be-47ef-b82b-abdacb12de8f","Type":"ContainerStarted","Data":"1993fd24b1ba8190fafaa8bfb39a9c8391dcb4db4e4180b0f58be2cdda85aeaa"} Nov 03 20:17:52.340521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.340450 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" podStartSLOduration=2.340429005 podStartE2EDuration="2.340429005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:52.336992284 +0000 UTC m=+1213.061999880" watchObservedRunningTime="2026-11-03 20:17:52.340429005 +0000 UTC m=+1213.065436601" Nov 03 20:17:52.600546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.600511 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:52.704927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:52.704893 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:53.039575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.039540 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j"] Nov 03 20:17:53.044913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.044870 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:53.045457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.045432 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a2449b-r2z9j"] Nov 03 20:17:53.370478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.369620 5954 generic.go:356] "Generic (PLEG): container finished" podID="0f6c1b34-e2be-47ef-b82b-abdacb12de8f" containerID="d4d4721323e00ac769e6a6e03df1294128ab589799ad8765fec0a0705dd8675a" exitCode=0 Nov 03 20:17:53.370478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.369730 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" event={"ID":"0f6c1b34-e2be-47ef-b82b-abdacb12de8f","Type":"ContainerDied","Data":"d4d4721323e00ac769e6a6e03df1294128ab589799ad8765fec0a0705dd8675a"} Nov 03 20:17:53.514781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.514746 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37990633-fb7c-4247-b58a-a63c8749f888" path="/var/lib/kubelet/pods/37990633-fb7c-4247-b58a-a63c8749f888/volumes" Nov 03 20:17:53.549195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.549160 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:53.777718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:53.777637 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.553490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.553464 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:54.579492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.579457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.619078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.619038 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\") pod \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " Nov 03 20:17:54.619291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.619203 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\" (UniqueName: \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\") pod \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " Nov 03 20:17:54.619291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.619242 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\") pod \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\" (UID: \"0f6c1b34-e2be-47ef-b82b-abdacb12de8f\") " Nov 03 20:17:54.620474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.620448 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca" pod "0f6c1b34-e2be-47ef-b82b-abdacb12de8f" (UID: "0f6c1b34-e2be-47ef-b82b-abdacb12de8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:54.653472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.653346 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq" pod "0f6c1b34-e2be-47ef-b82b-abdacb12de8f" (UID: "0f6c1b34-e2be-47ef-b82b-abdacb12de8f"). InnerVolumeSpecName "kube-api-access-flzrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:54.662074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.661304 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca" pod "0f6c1b34-e2be-47ef-b82b-abdacb12de8f" (UID: "0f6c1b34-e2be-47ef-b82b-abdacb12de8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:54.724997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.721826 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzrq\" (UniqueName: \"kubernetes.io/projected/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-kube-api-access-flzrq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:54.724997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.721863 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:54.724997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:54.721878 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6c1b34-e2be-47ef-b82b-abdacb12de8f-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:55.387940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:55.387904 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" event={"ID":"0f6c1b34-e2be-47ef-b82b-abdacb12de8f","Type":"ContainerDied","Data":"1993fd24b1ba8190fafaa8bfb39a9c8391dcb4db4e4180b0f58be2cdda85aeaa"} Nov 03 20:17:55.388129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:55.387950 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1993fd24b1ba8190fafaa8bfb39a9c8391dcb4db4e4180b0f58be2cdda85aeaa" Nov 03 20:17:55.388129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:55.387917 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd" Nov 03 20:17:55.560056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:55.560006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:56.355550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:56.355520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:56.480440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:56.480392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:56.589118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:56.588979 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:57.406117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:57.406068 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:57.792851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:57.792763 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:57.902803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:57.902743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:58.566054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:58.566017 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:59.400127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:17:59.399921 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:00.337228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:00.337178 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:00.415364 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:18:00.414902 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:18:00.501615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:00.501521 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:01.076836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:01.076800 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:01.743016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:01.742933 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:02.508212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:02.508179 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.291085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:03.291049 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.391246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:03.391210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.816447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:03.816406 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.026361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.026324 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt"] Nov 03 20:18:04.030814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.030786 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f331a05-t8lbt"] Nov 03 20:18:04.052212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.052179 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:18:04.053036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.053013 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0f6c1b34-e2be-47ef-b82b-abdacb12de8f" containerName="revoke-keys" Nov 03 20:18:04.053036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.053035 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6c1b34-e2be-47ef-b82b-abdacb12de8f" containerName="revoke-keys" Nov 03 20:18:04.053268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.053253 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6c1b34-e2be-47ef-b82b-abdacb12de8f" containerName="revoke-keys" Nov 03 20:18:04.057075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.057056 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.061662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.061639 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-1c6a98-serving-cert\"" Nov 03 20:18:04.068229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.068176 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:18:04.151039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.151001 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.151371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.151164 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.151371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.151345 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.151523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.151374 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8l9\" (UniqueName: \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.169151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.169092 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.230917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.230879 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:18:04.235802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.235771 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.251769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.251731 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:18:04.252965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.252937 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.253090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.253018 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.253701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.253191 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.253701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.253225 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8l9\" (UniqueName: \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.254237 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:18:04.254200 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-serving-cert: secret "maas-api-e2e-shared-b-w4-1c6a98-serving-cert" not found Nov 03 20:18:04.254428 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:18:04.254403 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls podName:a9718a5d-fbb7-4f03-bb92-f88f7b0715a6 nodeName:}" failed. No retries permitted until 2026-11-03 20:18:04.754389114 +0000 UTC m=+1225.479396692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls") pod "maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" (UID: "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6") : secret "maas-api-e2e-shared-b-w4-1c6a98-serving-cert" not found Nov 03 20:18:04.254536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.254267 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.256336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.256316 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.271157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.262372 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-ee4c78-openshift-default-dockercfg-tfq5b\"" Nov 03 20:18:04.271157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.269007 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8l9\" (UniqueName: \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.280573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.280519 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.355420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355384 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr548\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355539 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355604 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355684 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355707 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355759 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.355892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355888 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.356285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.355968 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.356285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.356025 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.376390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.376359 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458380 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458502 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458560 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458609 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458740 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.458934 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.459057 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.459532 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.459703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.459682 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.464142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.459919 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr548\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.502137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.491575 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.511125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.504411 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.511125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.505478 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.511125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.505722 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.616612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.578700 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.616612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.593663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.616612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.597828 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr548\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.616612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.599028 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\") pod \"e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.680440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.648490 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:04.778736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.776055 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.940184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.936736 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:04.984134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:04.982431 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:05.288726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.288681 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:05.466148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.457747 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6722e099-ad7d-4104-a921-d3d19bfdd3e0" path="/var/lib/kubelet/pods/6722e099-ad7d-4104-a921-d3d19bfdd3e0/volumes" Nov 03 20:18:05.788572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.776510 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:18:05.925207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.903664 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:18:05.945149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.934441 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:05.945149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.934556 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:05.945149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.934599 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:05.979895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.979856 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" event={"ID":"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6","Type":"ContainerStarted","Data":"69c9d37a4dd3fe2263bc3b66fecf036d5cae1f9d7255892c238f813c640bd025"} Nov 03 20:18:05.982646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:05.982617 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerStarted","Data":"680c0a89ac39c26d84d0d360dcef87eb3032c493f3f54f468a39b1b2f8903ce2"} Nov 03 20:18:06.144201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:06.144160 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:06.864694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:06.864663 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:06.993230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:06.993139 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerStarted","Data":"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35"} Nov 03 20:18:06.995572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:06.995534 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" event={"ID":"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6","Type":"ContainerStarted","Data":"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1"} Nov 03 20:18:06.995697 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:06.995595 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:07.019121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.019048 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podStartSLOduration=3.019002494 podStartE2EDuration="3.019002494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:07.01772597 +0000 UTC m=+1227.742733591" watchObservedRunningTime="2026-11-03 20:18:07.019002494 +0000 UTC m=+1227.744010089" Nov 03 20:18:07.045229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.045183 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" podStartSLOduration=2.485108098 podStartE2EDuration="3.045167429s" totalImagesPullingTime="560.059331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:07.041056717 +0000 UTC m=+1227.766064314" watchObservedRunningTime="2026-11-03 20:18:07.045167429 +0000 UTC m=+1227.770175027" Nov 03 20:18:07.057637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.057607 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:18:07.080871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.080839 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:07.663867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.663836 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:07.669448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.669422 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:07.978949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:07.978839 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:08.006992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:08.006965 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:08.008051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:08.008020 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:09.169506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:09.169476 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.381359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:09.381317 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.485768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:09.485687 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.986555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:09.986523 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.622987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:10.622948 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.731295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:10.731252 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.834516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:10.834486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.580702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:11.580665 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:12.381186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:12.381144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:13.012505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:13.012476 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:18:13.110875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:13.110832 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:13.935334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:13.935284 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:14.890906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:14.890868 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:15.601397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:15.601363 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:16.394073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:16.394041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.081208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:17.081176 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.782811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:17.782775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.912403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:17.911363 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:18.057411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:18.057276 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm"] Nov 03 20:18:18.070175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:18.070144 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d6db61-2vdrm"] Nov 03 20:18:18.792625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:18.792581 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:19.408791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:19.408761 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f570a69f-e77c-456a-8954-b578aebf950c" path="/var/lib/kubelet/pods/f570a69f-e77c-456a-8954-b578aebf950c/volumes" Nov 03 20:18:19.578178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:19.578146 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:20.368128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:20.368074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:21.002754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:21.002721 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:21.712998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:21.712957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:22.394746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:22.394690 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:23.089084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:23.089029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:23.807018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:23.806985 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:24.512861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:24.512814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:25.027834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:25.027801 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x"] Nov 03 20:18:25.031974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:25.031949 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0558c9f6-tfn7x"] Nov 03 20:18:25.284356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:25.284247 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:25.436759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:25.436709 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3bd205-2d06-4df7-a578-19df49be73d4" path="/var/lib/kubelet/pods/ed3bd205-2d06-4df7-a578-19df49be73d4/volumes" Nov 03 20:18:25.980037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:25.980002 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:26.658455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:26.658416 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:27.603995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:27.603962 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:28.480350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:28.480320 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:28.600646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:28.600610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:29.198416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:29.198386 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:29.470466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:29.470374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:29.989326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:29.989294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:30.260243 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:18:30.260162 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:18:30.668163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:30.668134 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.348420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:31.348390 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.984543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:31.984513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:32.138131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.134230 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:32.540831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.540798 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:18:32.545820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.545775 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.559710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.559680 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:18:32.560656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.560629 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-15e02e6b-openshift-default-dockercfg-mjzbc\"" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585400 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585464 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585490 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544pl\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585528 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585667 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585692 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585745 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585942 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.586482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.585979 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.589157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.589124 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801430 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801503 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801589 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801650 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801686 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544pl\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801732 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801833 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801861 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.801939 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.854425 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.855567 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.855845 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.856126 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.863194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.856742 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.911162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.888760 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:32.944558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.943434 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:32.953131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.952505 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:33.003142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:32.993282 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:33.020913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.015661 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:33.053341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.043267 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544pl\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:33.191220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.190474 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:33.600663 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:18:33.600630 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9945dcb3_d093_4f50_bcf3_465c5682c05b.slice/crio-7f048978868e6e3870e6ace76cb564e4cc67945b0e12c78caa75bbf1ffe6879a WatchSource:0}: Error finding container 7f048978868e6e3870e6ace76cb564e4cc67945b0e12c78caa75bbf1ffe6879a: Status 404 returned error can't find the container with id 7f048978868e6e3870e6ace76cb564e4cc67945b0e12c78caa75bbf1ffe6879a Nov 03 20:18:33.601003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.600811 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:18:33.603301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.603272 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:33.603375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.603336 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:33.603375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.603363 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:33.683973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:33.683919 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:34.498485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:34.498448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:34.638134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:34.631188 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerStarted","Data":"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503"} Nov 03 20:18:34.638134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:34.631324 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerStarted","Data":"7f048978868e6e3870e6ace76cb564e4cc67945b0e12c78caa75bbf1ffe6879a"} Nov 03 20:18:34.686727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:34.686664 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podStartSLOduration=2.68664413 podStartE2EDuration="2.68664413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:34.665752168 +0000 UTC m=+1255.390759765" watchObservedRunningTime="2026-11-03 20:18:34.68664413 +0000 UTC m=+1255.411651726" Nov 03 20:18:35.191848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:35.191823 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:35.197716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:35.197691 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:35.313001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:35.312970 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:35.639363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:35.639319 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:35.640356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:35.640332 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:18:36.082561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:36.082469 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:36.383732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:36.383701 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:18:36.944966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:36.944930 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:38.159136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:38.152560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:39.361130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:39.358733 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:40.742074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:40.741979 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:40.893129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:40.887557 5954 scope.go:122] "RemoveContainer" containerID="13336541bda4e8f40cf88a711b1b19abad65eae02262bb0196d0ec2c55ff7338" Nov 03 20:18:41.228905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.228871 5954 scope.go:122] "RemoveContainer" containerID="661239ced010ad38eb3b34262cf37507406f84a5c4cdc4bfc1aac459c531924e" Nov 03 20:18:41.316127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.311090 5954 scope.go:122] "RemoveContainer" containerID="97817a091b735ac3d184f402fd0e11b4f316345e5d281792163a4f239a42e93b" Nov 03 20:18:41.433131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.419345 5954 scope.go:122] "RemoveContainer" containerID="8481b61dda3b2efb3b790695c4cd2861afa7b46700068a56b5f0d8912c79d3ac" Nov 03 20:18:41.473189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.469484 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.540223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.540195 5954 scope.go:122] "RemoveContainer" containerID="198f262aabdb92689772531892557263e0d1e7e15de1530acd3e98633de2280c" Nov 03 20:18:41.688745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.688703 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.735387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.735298 5954 scope.go:122] "RemoveContainer" containerID="1672e6f974a5c751d15dd565bab82e2fa0c473ce50bd8275dc5e4252bebc8e11" Nov 03 20:18:41.756854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.755688 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.758623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.758184 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.886724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.886280 5954 scope.go:122] "RemoveContainer" containerID="1ce1c57e515b4d35f9edd902ea065726860f87f1c34d214fd3b1ceaa19f7be62" Nov 03 20:18:41.923475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.923441 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:41.967786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:41.967712 5954 scope.go:122] "RemoveContainer" containerID="e3d24ea454e271b0803645b3927ed6a2d03ed32e92341a1e11c215f3499e9af5" Nov 03 20:18:42.184130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.180410 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.184130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.183655 5954 scope.go:122] "RemoveContainer" containerID="c08d7d6c544f979e4eb090aca252f6ecde59932bb3917c4e7a0607406884b2c1" Nov 03 20:18:42.266215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.258513 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.338207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.332240 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.405132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.389346 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.483537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.475875 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.483537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.481497 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:18:42.704214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.699934 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.707537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.705967 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:18:42.827319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.827217 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.828721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.828684 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.836715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.836687 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:18:42.837051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:42.837033 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:18:43.221045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.221011 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:18:43.295209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.295130 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:43.295977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.295716 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:18:43.303640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.303612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:43.358218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.334773 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:18:43.362173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.361176 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:18:43.362173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.361466 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:18:43.437131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.436856 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:18:43.767160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.766925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:43.883627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:43.883590 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.027187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:44.011369 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.166137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:44.165673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.356631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:44.356600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.556726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:44.556697 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.938182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:44.932822 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:45.073131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:45.070158 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:45.207182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:45.206275 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:45.959123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:45.959074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:46.736804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.734760 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99"] Nov 03 20:18:46.742829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.742798 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99"] Nov 03 20:18:46.742990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.742910 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.746501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.746479 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:18:46.746642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.746540 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:18:46.872624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.872583 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.872818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.872641 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.872896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.872845 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grspt\" (UniqueName: \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.974789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.974746 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.974789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.974792 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.975329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.974851 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grspt\" (UniqueName: \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.975645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.975618 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.977412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.977385 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:46.987147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:46.987082 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grspt\" (UniqueName: \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\") pod \"maas-api-revoke-sub-e5b6be4b-zvn99\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:47.061177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:47.061147 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:47.087973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:47.087934 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:47.608377 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:18:47.608326 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270f7bc4_6cfe_477b_b353_f0a95ab82b5b.slice/crio-b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a WatchSource:0}: Error finding container b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a: Status 404 returned error can't find the container with id b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a Nov 03 20:18:47.609207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:47.609175 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99"] Nov 03 20:18:47.833816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:47.833777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.227984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.227879 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.487860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.487778 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" event={"ID":"270f7bc4-6cfe-477b-b353-f0a95ab82b5b","Type":"ContainerStarted","Data":"aedb7099d24399e109ff5d549e73737c5505dbe015b2dcffa0dcb604939a023a"} Nov 03 20:18:48.487860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.487839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" event={"ID":"270f7bc4-6cfe-477b-b353-f0a95ab82b5b","Type":"ContainerStarted","Data":"b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a"} Nov 03 20:18:48.506729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.506675 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" podStartSLOduration=2.506661371 podStartE2EDuration="2.506661371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:48.505889266 +0000 UTC m=+1269.230896863" watchObservedRunningTime="2026-11-03 20:18:48.506661371 +0000 UTC m=+1269.231668963" Nov 03 20:18:48.783557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.783491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.891750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:48.891716 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:49.161605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:49.161574 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:49.275004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:49.274974 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:49.595258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:49.592613 5954 generic.go:356] "Generic (PLEG): container finished" podID="270f7bc4-6cfe-477b-b353-f0a95ab82b5b" containerID="aedb7099d24399e109ff5d549e73737c5505dbe015b2dcffa0dcb604939a023a" exitCode=0 Nov 03 20:18:49.595258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:49.592862 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" event={"ID":"270f7bc4-6cfe-477b-b353-f0a95ab82b5b","Type":"ContainerDied","Data":"aedb7099d24399e109ff5d549e73737c5505dbe015b2dcffa0dcb604939a023a"} Nov 03 20:18:50.002080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:50.001974 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:50.106317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:50.106238 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:50.575889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:50.575849 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:50.961708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:50.961674 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:51.065877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.065840 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\" (UniqueName: \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\") pod \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " Nov 03 20:18:51.065877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.065876 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\") pod \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " Nov 03 20:18:51.066161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.066088 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\") pod \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\" (UID: \"270f7bc4-6cfe-477b-b353-f0a95ab82b5b\") " Nov 03 20:18:51.066536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.066454 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca" pod "270f7bc4-6cfe-477b-b353-f0a95ab82b5b" (UID: "270f7bc4-6cfe-477b-b353-f0a95ab82b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:51.066926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.066905 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:51.068155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.068083 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca" pod "270f7bc4-6cfe-477b-b353-f0a95ab82b5b" (UID: "270f7bc4-6cfe-477b-b353-f0a95ab82b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:51.068634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.068611 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt" pod "270f7bc4-6cfe-477b-b353-f0a95ab82b5b" (UID: "270f7bc4-6cfe-477b-b353-f0a95ab82b5b"). InnerVolumeSpecName "kube-api-access-grspt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:51.172089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.172014 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.191473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.191366 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grspt\" (UniqueName: \"kubernetes.io/projected/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-kube-api-access-grspt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:51.191473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.191398 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270f7bc4-6cfe-477b-b353-f0a95ab82b5b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:51.278018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.277982 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.481460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.473493 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.758178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.758007 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" event={"ID":"270f7bc4-6cfe-477b-b353-f0a95ab82b5b","Type":"ContainerDied","Data":"b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a"} Nov 03 20:18:51.758178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.758077 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f1791e847bf3ca9aa94d7fcd9f45da62224a4e32e055de7f413e23f369c98a" Nov 03 20:18:51.758178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.758161 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99" Nov 03 20:18:51.776131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.773456 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff04bd71-316c-473d-a766-220368ec68f4" containerID="ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35" exitCode=0 Nov 03 20:18:51.776131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.773568 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerDied","Data":"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35"} Nov 03 20:18:51.776131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.774258 5954 scope.go:122] "RemoveContainer" containerID="ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35" Nov 03 20:18:51.776832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.776772 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:51.776904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.776869 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:51.776962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:51.776918 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:52.438780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:52.438685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:52.783952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:52.783871 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerStarted","Data":"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807"} Nov 03 20:18:53.006998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:53.006958 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:53.262608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:53.262576 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:53.663719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:53.663677 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:53.664847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:53.664817 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:53.664991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:53.664874 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:54.045215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:54.045128 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:54.664135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:54.664085 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:54.664310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:54.664156 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:54.894995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:54.894920 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.641291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:55.641208 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.664990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:55.664950 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:55.665594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:55.665562 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:56.500788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:56.500756 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:56.603857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:56.603817 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:56.664985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:56.664532 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:56.664985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:56.664593 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:57.039650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:57.039608 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:57.278954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:57.278918 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:57.663921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:57.663884 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:57.664098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:57.663943 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:58.091639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.091561 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:58.663743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.663699 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:18:58.663963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.663780 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 20:18:58.760135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.760083 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:18:58.761911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.761884 5954 cpu_manager.go:414] "containerMap: removing container" podUID="270f7bc4-6cfe-477b-b353-f0a95ab82b5b" containerName="revoke-keys" Nov 03 20:18:58.762090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.762074 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270f7bc4-6cfe-477b-b353-f0a95ab82b5b" containerName="revoke-keys" Nov 03 20:18:58.762496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.762477 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="270f7bc4-6cfe-477b-b353-f0a95ab82b5b" containerName="revoke-keys" Nov 03 20:18:58.776117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.776078 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:18:58.776305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.776195 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.781537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.781480 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7ea43c-serving-cert\"" Nov 03 20:18:58.785514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.785489 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.785666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.785647 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc296\" (UniqueName: \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.785932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.785898 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.786096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.786071 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.800803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.800764 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:58.887747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.887710 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.887960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.887828 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.888034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.887961 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.888034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.887997 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc296\" (UniqueName: \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.888561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.888527 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.890273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.890253 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.890512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.890492 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:58.899958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.899931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:58.900454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:58.900430 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc296\" (UniqueName: \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\") pod \"maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:59.108559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.107328 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:18:59.509014 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:18:59.508963 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda062c118_5e7a_46dd_8399_80b3a7b3c986.slice/crio-a1a6900835d8ba578998b878286627c773ab065d887997d668b1428ae85fa7be WatchSource:0}: Error finding container a1a6900835d8ba578998b878286627c773ab065d887997d668b1428ae85fa7be: Status 404 returned error can't find the container with id a1a6900835d8ba578998b878286627c773ab065d887997d668b1428ae85fa7be Nov 03 20:18:59.540528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.534631 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:18:59.685445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.685411 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:59.695432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.695404 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:18:59.762243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.762158 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:18:59.770592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.770560 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:18:59.776681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.776651 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-15e02e6b-serving-cert\"" Nov 03 20:18:59.790226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.781444 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:18:59.827652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.827621 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:59.920237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.919732 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:18:59.920237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.919942 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:18:59.920237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.920022 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:18:59.920237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.920203 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzgk\" (UniqueName: \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:18:59.964840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:18:59.964755 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.025177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.018745 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" event={"ID":"a062c118-5e7a-46dd-8399-80b3a7b3c986","Type":"ContainerStarted","Data":"a1a6900835d8ba578998b878286627c773ab065d887997d668b1428ae85fa7be"} Nov 03 20:19:00.041436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.029121 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.041436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.029483 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzgk\" (UniqueName: \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.041436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.040033 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.041436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.040303 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.041436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.041351 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.083128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.071906 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.104916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.088821 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzgk\" (UniqueName: \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.104916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.103679 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.106043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.105982 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj"] Nov 03 20:19:00.128129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.113989 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.180130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.179535 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f443ba3b-xx2vj"] Nov 03 20:19:00.450130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.407208 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:00.557551 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:19:00.557513 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:19:00.871918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:00.871878 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:01.133643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.132782 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" event={"ID":"a062c118-5e7a-46dd-8399-80b3a7b3c986","Type":"ContainerStarted","Data":"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d"} Nov 03 20:19:01.133643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.133214 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:19:01.167169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.167084 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" podStartSLOduration=2.436217421 podStartE2EDuration="3.167064811s" totalImagesPullingTime="730.84739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:01.15861497 +0000 UTC m=+1281.883622566" watchObservedRunningTime="2026-11-03 20:19:01.167064811 +0000 UTC m=+1281.892072409" Nov 03 20:19:01.210914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.210863 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:19:01.211562 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:19:01.211536 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6feb82_f483_403f_885c_09881e53bae5.slice/crio-823197f7506d36cea0048d72bf1f3ef924cad47bd92d08d3496c1a40cef37b29 WatchSource:0}: Error finding container 823197f7506d36cea0048d72bf1f3ef924cad47bd92d08d3496c1a40cef37b29: Status 404 returned error can't find the container with id 823197f7506d36cea0048d72bf1f3ef924cad47bd92d08d3496c1a40cef37b29 Nov 03 20:19:01.446813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.446737 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48363c75-9a78-4f0d-af80-5bf9f9276532" path="/var/lib/kubelet/pods/48363c75-9a78-4f0d-af80-5bf9f9276532/volumes" Nov 03 20:19:01.908291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:01.906584 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:02.042968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.042932 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z"] Nov 03 20:19:02.070512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.070471 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7031aae6-pn97z"] Nov 03 20:19:02.168331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.168148 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" event={"ID":"7e6feb82-f483-403f-885c-09881e53bae5","Type":"ContainerStarted","Data":"716546d420d1ab1a990348d27a204cd9ef3868137e3845609d350c4bad33c23c"} Nov 03 20:19:02.168331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.168204 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" event={"ID":"7e6feb82-f483-403f-885c-09881e53bae5","Type":"ContainerStarted","Data":"823197f7506d36cea0048d72bf1f3ef924cad47bd92d08d3496c1a40cef37b29"} Nov 03 20:19:02.168331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.168250 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:02.192857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:02.192748 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" podStartSLOduration=2.565493479 podStartE2EDuration="3.192728719s" totalImagesPullingTime="627.23524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:02.191466968 +0000 UTC m=+1282.916474565" watchObservedRunningTime="2026-11-03 20:19:02.192728719 +0000 UTC m=+1282.917736316" Nov 03 20:19:03.001126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:03.000754 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:03.414141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:03.414054 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8eef69-d411-4a94-96db-db7bb1ab4e9a" path="/var/lib/kubelet/pods/0a8eef69-d411-4a94-96db-db7bb1ab4e9a/volumes" Nov 03 20:19:03.785709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:03.785616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:04.284821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:04.284670 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:04.413580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:04.404343 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:05.789670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:05.789628 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:06.590725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:06.590696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.033643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:07.033608 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp"] Nov 03 20:19:07.038132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:07.038091 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7fa66d1-5d4fp"] Nov 03 20:19:07.172386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:07.172354 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:19:07.285652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:07.285573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.505713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:07.505678 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eccac9-5a79-41d2-bf90-42fd836b01e6" path="/var/lib/kubelet/pods/b4eccac9-5a79-41d2-bf90-42fd836b01e6/volumes" Nov 03 20:19:08.012846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:08.012809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:08.705399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:08.705347 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:09.305529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:09.305499 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:19:09.439553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:09.439520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:10.185612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:10.185551 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:10.865261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:10.865226 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:11.573892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:11.573851 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:12.755609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:12.755562 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:13.767990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:13.767916 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:13.973838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:13.973718 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.316358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:14.316319 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.598231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:14.598150 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.699878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:14.699836 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:15.088291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:15.088257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:15.563507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:15.563469 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:15.795336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:15.790659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.325593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:16.325556 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.790205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:16.790173 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:17.676173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:17.676134 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:18.662563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:18.657816 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:18.976278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:18.976185 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:19.080571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:19.080513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:19.404508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:19.404478 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:20.190386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:20.190347 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:20.298669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:20.298614 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.022060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:21.022025 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.382458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:21.382423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.585508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:21.585472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:22.494130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:22.494069 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:23.457424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:23.457389 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:24.256421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:24.256380 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:24.603344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:24.603316 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:24.713252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:24.713207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:25.122250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:25.122220 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:25.798531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:25.798484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:25.916122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:25.916075 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:26.087644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:26.087548 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:26.967216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:26.967183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:27.650916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:27.650879 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:27.790186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:27.780071 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:28.027722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:28.027644 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz"] Nov 03 20:19:28.032393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:28.032369 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f11dd46-9dbdz"] Nov 03 20:19:28.402369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:28.402337 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:29.180835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:29.180799 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:29.408870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:29.408840 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16417dc3-b680-4459-afba-f8abfc7e42fc" path="/var/lib/kubelet/pods/16417dc3-b680-4459-afba-f8abfc7e42fc/volumes" Nov 03 20:19:29.916847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:29.916780 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.259725 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:19:30.259640 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:19:30.749754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.749701 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:19:30.757168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.757143 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:30.762636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.762611 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:19:30.764253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.764231 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9af0cd-serving-cert\"" Nov 03 20:19:30.786061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.786029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.933004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.932969 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.947159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.936974 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vhw\" (UniqueName: \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:30.947159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.937362 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:30.947159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.937510 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:30.947159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:30.937537 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.041638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.041542 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.041938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.041916 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.042062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.042045 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.042382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.042354 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vhw\" (UniqueName: \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.062233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.062198 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vhw\" (UniqueName: \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.066150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.065478 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.066150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.066083 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.067446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.067403 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\") pod \"maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.078546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.078518 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:31.229400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.229356 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:19:31.236848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.236750 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.249153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.248353 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:19:31.250577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.250549 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06294c6a-openshift-default-dockercfg-b69h6\"" Nov 03 20:19:31.254616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.254593 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:31.491152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.422973 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.843817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843395 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.843817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843443 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.843817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843497 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.843817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843545 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2svw\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.843817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843682 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.844194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843949 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.844194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.843983 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.844194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.844041 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.869128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.858599 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965439 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965490 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965539 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965578 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2svw\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965687 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965884 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965913 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.965960 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.966019 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.980302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.966912 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.981778 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.982041 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.983512 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.983828 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.983989 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:31.988941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:31.985252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:32.045880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.045773 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:32.097475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.070806 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2svw\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\") pod \"e2e-mt-06294c6a-openshift-default-85555b884f-67qxj\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:32.135232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.124930 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:32.177272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.168563 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:32.177272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.169048 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerStarted","Data":"66ea018a5c7d24f8fa49616c1050c75127c9d25fd404de4340ed6399c4c33f5f"} Nov 03 20:19:32.300141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.294374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:32.960866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.960840 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:19:32.983339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.979483 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:32.983339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.979568 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:32.983339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:32.979620 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:33.029134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.028826 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6"] Nov 03 20:19:33.035715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.035676 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0390c9ab-xmrz6"] Nov 03 20:19:33.107260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.107231 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:33.227179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.221813 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerStarted","Data":"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99"} Nov 03 20:19:33.227179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.224644 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:33.244160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.236136 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerStarted","Data":"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769"} Nov 03 20:19:33.244160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.236203 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerStarted","Data":"9aabd60bfc5ade4b5501d89270305b7b3d37205a8501ebbf835dd119b5ac1227"} Nov 03 20:19:33.278128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.276439 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" podStartSLOduration=2.677214304 podStartE2EDuration="3.276417941s" totalImagesPullingTime="599.203637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:33.247005257 +0000 UTC m=+1313.972012866" watchObservedRunningTime="2026-11-03 20:19:33.276417941 +0000 UTC m=+1314.001425536" Nov 03 20:19:33.341680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.341615 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podStartSLOduration=2.341592366 podStartE2EDuration="2.341592366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:33.305267853 +0000 UTC m=+1314.030275447" watchObservedRunningTime="2026-11-03 20:19:33.341592366 +0000 UTC m=+1314.066599961" Nov 03 20:19:33.413663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:33.411562 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151fee40-ffd7-4c60-a990-5de55fc39cad" path="/var/lib/kubelet/pods/151fee40-ffd7-4c60-a990-5de55fc39cad/volumes" Nov 03 20:19:34.120169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.120136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:34.169351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.169317 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:34.175368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.175344 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:34.222142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.221611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:34.289169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.289139 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:34.291028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:34.291004 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:19:35.173317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:35.173281 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:35.667930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:35.667893 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:35.768014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:35.767980 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:36.053373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:36.053290 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:36.827337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:36.827306 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:36.940686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:36.940641 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:37.467034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:37.464591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:37.564336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:37.564019 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:37.691176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:37.689931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:38.447398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:38.447368 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:39.311630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:39.311597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:39.885066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:39.885033 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:19:40.244213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:40.244056 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:40.352601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:40.352565 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.133:8443/health\": dial tcp 10.134.0.133:8443: connect: connection refused" Nov 03 20:19:41.228135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:41.222595 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:41.589181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:41.583366 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:42.799065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:42.799024 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:43.519133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.512549 5954 scope.go:122] "RemoveContainer" containerID="4b17145cddc673542c78c04fee409e151e2a4bf0c170c7440a2639d386220980" Nov 03 20:19:43.588174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.588144 5954 scope.go:122] "RemoveContainer" containerID="2081314dfb3b493f3cd2970a8f24dc81b13351f89dbb2a34fdf79bee159e3e7d" Nov 03 20:19:43.605270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.605084 5954 scope.go:122] "RemoveContainer" containerID="f0bf4dc71f27ae278234f13bd8d61d21cc0b631754fa703319fdc8f306b70c71" Nov 03 20:19:43.663744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.663276 5954 scope.go:122] "RemoveContainer" containerID="03776683a1715db59842471e8742bedd0aa6b439cc4f5300046e240905b50a74" Nov 03 20:19:43.862503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.811702 5954 scope.go:122] "RemoveContainer" containerID="be40508cde0ca2fafc238ef1451afe8a1c80034fc2b76601c9f7e9e1e0251e9b" Nov 03 20:19:43.862503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.829618 5954 generic.go:356] "Generic (PLEG): container finished" podID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerID="ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99" exitCode=1 Nov 03 20:19:43.862503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.829801 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerDied","Data":"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99"} Nov 03 20:19:43.862503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:43.830372 5954 scope.go:122] "RemoveContainer" containerID="ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99" Nov 03 20:19:44.251820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:44.251780 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:45.192128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:45.192041 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.192614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:45.192427 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.476131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:45.475592 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.640267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:45.640223 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.853861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:45.853823 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:46.020133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:46.012553 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerStarted","Data":"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf"} Nov 03 20:19:46.020133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:46.014143 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:47.811397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:47.811358 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.921130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:47.914058 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:48.587841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:48.587807 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:49.500148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:49.500079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.024225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:50.024193 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht"] Nov 03 20:19:50.028379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:50.028352 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34d0897a-bwqht"] Nov 03 20:19:50.292234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:50.292144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:51.079966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:51.079932 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:51.409336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:51.409307 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99c54ac-01a4-4b16-b2e4-d3f6f5421b92" path="/var/lib/kubelet/pods/e99c54ac-01a4-4b16-b2e4-d3f6f5421b92/volumes" Nov 03 20:19:51.732863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:51.732773 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.026092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.026015 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:19:52.447585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.447554 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:19:52.453404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.453381 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.457927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.457453 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06294c6a-serving-cert\"" Nov 03 20:19:52.463874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.463844 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:19:52.539218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.539187 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.550540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.550472 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.550870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.550818 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.550984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.550962 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.551300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.551276 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2r4\" (UniqueName: \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.658129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.654855 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.658129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.654932 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.658129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.655133 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2r4\" (UniqueName: \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.658129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.655292 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.679127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.674870 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.679127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.675657 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.700198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.692335 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.724330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.722787 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2r4\" (UniqueName: \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.724330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.723396 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\") pod \"maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:52.770299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.770249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.787903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:52.787866 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:53.460552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:53.418434 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:19:53.546888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:53.546848 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:54.377478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:54.377374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:54.556145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:54.556083 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" event={"ID":"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb","Type":"ContainerStarted","Data":"6199ed590476b57240b585cab845b8f24197f877ffb8986344eda332e64462f8"} Nov 03 20:19:54.556797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:54.556771 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" event={"ID":"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb","Type":"ContainerStarted","Data":"a15dce9459824e8a6e94adc250e301142f5b83a7dbe8927e30a310a265bed60c"} Nov 03 20:19:54.558513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:54.558491 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:19:55.245138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:55.245080 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.390042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:56.390009 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:57.026783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.026728 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" podStartSLOduration=4.405712629 podStartE2EDuration="5.026712484s" totalImagesPullingTime="620.999855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:54.590716298 +0000 UTC m=+1335.315723894" watchObservedRunningTime="2026-11-03 20:19:57.026712484 +0000 UTC m=+1337.751720123" Nov 03 20:19:57.030425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.030398 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82"] Nov 03 20:19:57.034742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.034720 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439024a5-q2k82"] Nov 03 20:19:57.188597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.188564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:57.408132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.407580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:57.488059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:57.488022 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc91c4d2-914a-477d-a9ab-1f4039fb6b4c" path="/var/lib/kubelet/pods/bc91c4d2-914a-477d-a9ab-1f4039fb6b4c/volumes" Nov 03 20:19:58.161099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:58.161062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:58.981214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:58.981174 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:59.719663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:19:59.719626 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:00.269875 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:20:00.269829 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:20:01.605431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:01.605400 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:20:02.391480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:02.391438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.488224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:02.488189 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.610562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:02.610519 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.980684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:02.980640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:03.098065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:03.097984 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:03.764934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:03.764901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:04.947418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:04.947383 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:05.880591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:05.880557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:06.238525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:06.238452 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:06.335597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:06.335564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:06.947272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:06.947245 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:07.060382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.060186 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx"] Nov 03 20:20:07.094986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.094906 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ebb2b7f-wkvlx"] Nov 03 20:20:07.413705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.413670 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c4c14f-5110-4813-a35d-fd6d54655558" path="/var/lib/kubelet/pods/28c4c14f-5110-4813-a35d-fd6d54655558/volumes" Nov 03 20:20:07.453310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.453272 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:07.565155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.565093 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:07.942871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:07.942834 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.061306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:09.061279 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.970310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:09.970268 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.077348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:10.077316 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.670879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:10.670842 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.781622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:10.781586 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:11.243172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:11.243138 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:11.395880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:11.395841 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.139185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:12.139154 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.242485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:12.242453 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.369131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:12.366514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.848336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:12.848306 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:13.275652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:13.275616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:14.120719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:14.120682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:14.876126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:14.876079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:15.574841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:15.574804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:16.299221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:16.299187 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:17.183371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:17.183313 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:17.926674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:17.926640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:18.722224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:18.722177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:19.073595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:19.073494 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:19.723550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:19.723509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.782128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:20.781551 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.987524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:20.975058 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:21.110671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:21.110635 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:21.261261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:21.256302 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq"] Nov 03 20:20:21.395685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:21.390860 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50a42809-fqfjq"] Nov 03 20:20:21.493881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:21.493846 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9269629c-101c-45e0-bda0-35965748e13e" path="/var/lib/kubelet/pods/9269629c-101c-45e0-bda0-35965748e13e/volumes" Nov 03 20:20:22.173136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:22.166224 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:23.166142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:23.166083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:23.797165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:23.797066 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:23.898854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:23.898806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:24.106196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:24.103417 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:24.884144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:24.884094 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:25.612382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:25.612345 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.374639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:26.374606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.772021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:26.771906 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.883204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:26.883172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.987870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:26.987838 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.292397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:27.292342 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.608533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:27.601743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:28.070272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:28.070187 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk"] Nov 03 20:20:28.080897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:28.080861 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0721ec-gfkvk"] Nov 03 20:20:28.083388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:28.083356 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:29.012512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:29.012465 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:29.416996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:29.416960 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f9e244-296d-4e4f-ba75-3fd0b3717d46" path="/var/lib/kubelet/pods/f0f9e244-296d-4e4f-ba75-3fd0b3717d46/volumes" Nov 03 20:20:29.894463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:29.894419 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:30.260055 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:20:30.259971 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:20:30.581013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:30.580924 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:31.239165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:31.239083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.018692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.018664 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.030826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.030797 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:20:32.031025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.031004 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" containerID="cri-o://7037319c30b55726056e2cf42933f70025ecf8ab55faeb047fc28af6fe113947" gracePeriod=30 Nov 03 20:20:32.136325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.110169 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:20:32.136325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.110503 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" containerID="cri-o://a03acdb717b31d3131d6ca70386b8445aeacf0a03926c069dad89269516e55cb" gracePeriod=30 Nov 03 20:20:32.273224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.273140 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:20:32.273453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.273397 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" containerID="cri-o://7c19cbfd7c198297885c8761777562728bfe11487f243b7c80c90f580530250e" gracePeriod=30 Nov 03 20:20:32.691765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.691721 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:20:32.691972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.691935 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" containerID="cri-o://70440dbd7097774ad601a92b3b551329a90fa299c269abe4688a46f689a2c020" gracePeriod=30 Nov 03 20:20:32.697478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.697447 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:20:32.697885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.697858 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" containerID="cri-o://8d8d0ce0e91b9ceb6b434386e179cbee06db31c16e3115c28e46eb2c2bf4983b" gracePeriod=30 Nov 03 20:20:32.986099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:32.986019 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:33.686613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:33.686583 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:34.253574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.253484 5954 generic.go:356] "Generic (PLEG): container finished" podID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerID="1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503" exitCode=0 Nov 03 20:20:34.253745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.253587 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerDied","Data":"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503"} Nov 03 20:20:34.254122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.254085 5954 scope.go:122] "RemoveContainer" containerID="1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503" Nov 03 20:20:34.255543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.255519 5954 generic.go:356] "Generic (PLEG): container finished" podID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerID="d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769" exitCode=0 Nov 03 20:20:34.255620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.255586 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerDied","Data":"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769"} Nov 03 20:20:34.255922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.255907 5954 scope.go:122] "RemoveContainer" containerID="d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769" Nov 03 20:20:34.255979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.255957 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.256035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.256023 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.256075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.256059 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.257452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.257424 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.257568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.257502 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.257630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.257565 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:34.289859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.289833 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:20:34.387065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:34.387026 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.155258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:35.151837 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.352597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:35.352551 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerStarted","Data":"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9"} Nov 03 20:20:35.367255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:35.358277 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerStarted","Data":"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8"} Nov 03 20:20:35.652225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:35.643295 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:20:36.052178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.051643 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf"] Nov 03 20:20:36.092132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.085522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.114150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.109423 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3be034e-8xjwf"] Nov 03 20:20:36.175130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.174474 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:20:36.175130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.174593 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:36.175130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.174636 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:36.194141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.192891 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:20:36.194141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.193053 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:36.194141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.193099 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:36.685128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.681933 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.988881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:36.988806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:37.195026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.193698 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:37.195026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.193772 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:37.195026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.194253 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:37.195026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.194329 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:37.480706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.480682 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1396f0-a7c9-4c84-b169-a359a01fc45b" path="/var/lib/kubelet/pods/4d1396f0-a7c9-4c84-b169-a359a01fc45b/volumes" Nov 03 20:20:37.580887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.580632 5954 generic.go:356] "Generic (PLEG): container finished" podID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerID="7037319c30b55726056e2cf42933f70025ecf8ab55faeb047fc28af6fe113947" exitCode=0 Nov 03 20:20:37.580887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.580839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" event={"ID":"b8ab4577-de9d-4a19-8e00-9f546754719c","Type":"ContainerDied","Data":"7037319c30b55726056e2cf42933f70025ecf8ab55faeb047fc28af6fe113947"} Nov 03 20:20:37.585145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.584156 5954 generic.go:356] "Generic (PLEG): container finished" podID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerID="7c19cbfd7c198297885c8761777562728bfe11487f243b7c80c90f580530250e" exitCode=0 Nov 03 20:20:37.585145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.584309 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" event={"ID":"953aca2f-55e6-4c11-b3d7-9a90a842b955","Type":"ContainerDied","Data":"7c19cbfd7c198297885c8761777562728bfe11487f243b7c80c90f580530250e"} Nov 03 20:20:37.587536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.587482 5954 generic.go:356] "Generic (PLEG): container finished" podID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerID="a03acdb717b31d3131d6ca70386b8445aeacf0a03926c069dad89269516e55cb" exitCode=0 Nov 03 20:20:37.587715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.587649 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" event={"ID":"0bc68032-d375-4667-84b7-59389b5fb9ee","Type":"ContainerDied","Data":"a03acdb717b31d3131d6ca70386b8445aeacf0a03926c069dad89269516e55cb"} Nov 03 20:20:37.882943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.882352 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:37.887369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.887339 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:20:37.893988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:37.893963 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060547 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060640 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060682 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060711 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060797 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060837 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060918 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060974 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.060999 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061066 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061160 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061193 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061249 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061340 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061367 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\") pod \"0bc68032-d375-4667-84b7-59389b5fb9ee\" (UID: \"0bc68032-d375-4667-84b7-59389b5fb9ee\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061437 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061605 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.064183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.061676 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\") pod \"b8ab4577-de9d-4a19-8e00-9f546754719c\" (UID: \"b8ab4577-de9d-4a19-8e00-9f546754719c\") " Nov 03 20:20:38.098465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.097579 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.098465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.097864 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.098465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.098074 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.174327 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.175305 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.175560 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.177523 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.177569 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.192192 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.192590 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.226513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.200201 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.250795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.228744 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.250795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.232630 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "kube-api-access-6rhgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:38.278204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.251352 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:38.279893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.279846 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282810 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282844 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282860 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282876 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282892 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8ab4577-de9d-4a19-8e00-9f546754719c-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282907 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282921 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282935 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282950 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282964 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rhgf\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-kube-api-access-6rhgf\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282979 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.288136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.282993 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc68032-d375-4667-84b7-59389b5fb9ee-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.317218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.311927 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:38.317218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.312006 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:38.424693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.422242 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:38.424693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.424247 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:38.476130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.475713 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:38.505136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.476934 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token" pod "0bc68032-d375-4667-84b7-59389b5fb9ee" (UID: "0bc68032-d375-4667-84b7-59389b5fb9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:38.544965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.544476 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.544965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.544508 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.544965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.544521 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc68032-d375-4667-84b7-59389b5fb9ee-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.544965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.544535 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:38.806132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.761884 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.129256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.902385 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp" pod "b8ab4577-de9d-4a19-8e00-9f546754719c" (UID: "b8ab4577-de9d-4a19-8e00-9f546754719c"). InnerVolumeSpecName "kube-api-access-4b8jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:39.129256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:38.902572 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:39.129256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.111753 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8ab4577-de9d-4a19-8e00-9f546754719c-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.129256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.111786 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8jp\" (UniqueName: \"kubernetes.io/projected/b8ab4577-de9d-4a19-8e00-9f546754719c-kube-api-access-4b8jp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.221386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.191339 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:39.221386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.191402 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:39.221386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.204714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:39.221386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.204931 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:20:39.261170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.252676 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:39.261170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.252749 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:39.261170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.253229 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz"] Nov 03 20:20:39.261170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.259368 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:20:39.261907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.261774 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" event={"ID":"b8ab4577-de9d-4a19-8e00-9f546754719c","Type":"ContainerDied","Data":"1f13a6d7cc9cd9676be78ad612acf36acbf3f6b00fefd6d5e85e1b6ae9e901eb"} Nov 03 20:20:39.261907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.261860 5954 scope.go:122] "RemoveContainer" containerID="7037319c30b55726056e2cf42933f70025ecf8ab55faeb047fc28af6fe113947" Nov 03 20:20:39.262128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.262070 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8" Nov 03 20:20:39.275157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.272225 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:39.291414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.288254 5954 generic.go:356] "Generic (PLEG): container finished" podID="2b57a418-15d9-488d-9007-9647debd6e6f" containerID="8d8d0ce0e91b9ceb6b434386e179cbee06db31c16e3115c28e46eb2c2bf4983b" exitCode=0 Nov 03 20:20:39.291414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.288487 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" event={"ID":"2b57a418-15d9-488d-9007-9647debd6e6f","Type":"ContainerDied","Data":"8d8d0ce0e91b9ceb6b434386e179cbee06db31c16e3115c28e46eb2c2bf4983b"} Nov 03 20:20:39.291414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.290572 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" event={"ID":"0bc68032-d375-4667-84b7-59389b5fb9ee","Type":"ContainerDied","Data":"54f8fc005de91df98a65326707ca2a0ba978d821d5273e7c9db9608ea88ca76a"} Nov 03 20:20:39.291414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.290673 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.300518 5954 generic.go:356] "Generic (PLEG): container finished" podID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerID="70440dbd7097774ad601a92b3b551329a90fa299c269abe4688a46f689a2c020" exitCode=0 Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.300586 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" event={"ID":"16a41eb9-59df-42a4-81eb-4413e5b204a2","Type":"ContainerDied","Data":"70440dbd7097774ad601a92b3b551329a90fa299c269abe4688a46f689a2c020"} Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.355977 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.374070 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.375006 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.376696 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.380652 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.380718 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.380764 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.381774 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.381882 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.385073 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.385182 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.385332 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.385405 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\") pod \"953aca2f-55e6-4c11-b3d7-9a90a842b955\" (UID: \"953aca2f-55e6-4c11-b3d7-9a90a842b955\") " Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.388915 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.390267 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.393805 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "kube-api-access-24nv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.396781 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.396978 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.423809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.423627 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:20:39.424962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.424449 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:39.424962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.424526 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:39.424962 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:20:39.424758 5954 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-41d22c1a-mcdxz\" is forbidden: User \"system:node:ip-10-0-135-70.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-70.us-east-2.compute.internal' and this object" podUID="e3e0b48e-8d31-489e-8759-281e75132e4e" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz" Nov 03 20:20:39.463575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.461611 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.536293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.505592 5954 scope.go:122] "RemoveContainer" containerID="a03acdb717b31d3131d6ca70386b8445aeacf0a03926c069dad89269516e55cb" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564017 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564055 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564074 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564092 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564127 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564315 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nv6\" (UniqueName: \"kubernetes.io/projected/953aca2f-55e6-4c11-b3d7-9a90a842b955-kube-api-access-24nv6\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564532 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953aca2f-55e6-4c11-b3d7-9a90a842b955-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.564726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.564551 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.567888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.567808 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:20:39.579011 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.578927 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy" pod "953aca2f-55e6-4c11-b3d7-9a90a842b955" (UID: "953aca2f-55e6-4c11-b3d7-9a90a842b955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.580015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.579991 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d22c1a-mcdxz"] Nov 03 20:20:39.580187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.580049 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599679 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599705 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599728 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599736 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599750 5954 cpu_manager.go:414] "containerMap: removing container" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599758 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599810 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.599818 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.601255 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.601311 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.601323 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.601335 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" containerName="istio-proxy" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.604959 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.605008 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.605096 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.609597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.609272 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-pzwh6"] Nov 03 20:20:39.645441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.645360 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2d5927-openshift-default-dockercfg-xbrcz\"" Nov 03 20:20:39.703493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.703018 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.703493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.703083 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.703493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.703197 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.704976 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705047 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705089 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705176 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705651 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705736 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705898 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.705964 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706024 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706176 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706384 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706464 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66775\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706604 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706649 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706713 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706887 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706905 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b57a418-15d9-488d-9007-9647debd6e6f-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.707263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.706919 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953aca2f-55e6-4c11-b3d7-9a90a842b955-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:39.708561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.708532 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.708969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.708949 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.710698 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "kube-api-access-rjhcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.712397 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.928143 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.954282 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.927839 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:39.959280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.954936 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\") pod \"2b57a418-15d9-488d-9007-9647debd6e6f\" (UID: \"2b57a418-15d9-488d-9007-9647debd6e6f\") " Nov 03 20:20:40.023744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:39.954357 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:20:40.083971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.083933 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-g8nl8"] Nov 03 20:20:40.087839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.087791 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:40.091035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.091010 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.091295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.091275 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66775\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.091519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.091505 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095285 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095356 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095394 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095570 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095631 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.095766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.095746 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.096041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096021 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjhcq\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-kube-api-access-rjhcq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096050 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096066 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096081 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b57a418-15d9-488d-9007-9647debd6e6f-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096094 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096124 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b57a418-15d9-488d-9007-9647debd6e6f-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.096465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.096443 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.097877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.097032 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token" pod "2b57a418-15d9-488d-9007-9647debd6e6f" (UID: "2b57a418-15d9-488d-9007-9647debd6e6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:40.097877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.097161 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.097877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.097417 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.097877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.097810 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.101867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.097874 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.101867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.099401 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.101867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.101581 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.105181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.105083 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.105291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.105271 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66775\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\") pod \"e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.181810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.175884 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:40.181810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.175947 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:40.192413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.192302 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:40.192413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.192364 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:40.198832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.198397 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b57a418-15d9-488d-9007-9647debd6e6f-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.250909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.250473 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:20:40.299331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299300 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299349 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299393 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299489 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299515 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299536 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299562 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299634 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.299691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.299666 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\") pod \"16a41eb9-59df-42a4-81eb-4413e5b204a2\" (UID: \"16a41eb9-59df-42a4-81eb-4413e5b204a2\") " Nov 03 20:20:40.300508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.300485 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:40.301279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.301254 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:40.301784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.301756 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:40.302039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.302014 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:40.302226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.302205 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:40.303435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.303412 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "kube-api-access-4c47r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:40.303920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.303891 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:40.304017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.303965 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:40.304522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.304500 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token" pod "16a41eb9-59df-42a4-81eb-4413e5b204a2" (UID: "16a41eb9-59df-42a4-81eb-4413e5b204a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:40.312859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.312834 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" event={"ID":"2b57a418-15d9-488d-9007-9647debd6e6f","Type":"ContainerDied","Data":"d6fb5befd2e6f0aeed73ce0d7e268e8b0d5e6f048cb543a38253f55cc639bcfa"} Nov 03 20:20:40.312950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.312928 5954 scope.go:122] "RemoveContainer" containerID="8d8d0ce0e91b9ceb6b434386e179cbee06db31c16e3115c28e46eb2c2bf4983b" Nov 03 20:20:40.313063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.313048 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8" Nov 03 20:20:40.379003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.378317 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:40.406866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.406820 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.411296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.411271 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.411903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.411876 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16a41eb9-59df-42a4-81eb-4413e5b204a2-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412030 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412184 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412343 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16a41eb9-59df-42a4-81eb-4413e5b204a2-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412463 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c47r\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-kube-api-access-4c47r\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412557 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412649 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.412637 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16a41eb9-59df-42a4-81eb-4413e5b204a2-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:40.412739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.408838 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" event={"ID":"953aca2f-55e6-4c11-b3d7-9a90a842b955","Type":"ContainerDied","Data":"187e6a36ea6301da80912a0cb9ad6dc55109ac1af9442b944f5f46fe62b92217"} Nov 03 20:20:40.412843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.408934 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5" Nov 03 20:20:40.475235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.475195 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:20:40.481140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.481083 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" event={"ID":"16a41eb9-59df-42a4-81eb-4413e5b204a2","Type":"ContainerDied","Data":"fd4aa2f96a7dba5805f46b29a843a0bc1df946daebae6144a2efc74651f22005"} Nov 03 20:20:40.481277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.481192 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-cdkp8"] Nov 03 20:20:40.481412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.481372 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx" Nov 03 20:20:40.488846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.488784 5954 scope.go:122] "RemoveContainer" containerID="7c19cbfd7c198297885c8761777562728bfe11487f243b7c80c90f580530250e" Nov 03 20:20:40.504380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.504285 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:20:40.507527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.506307 5954 scope.go:122] "RemoveContainer" containerID="70440dbd7097774ad601a92b3b551329a90fa299c269abe4688a46f689a2c020" Nov 03 20:20:40.559139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.537646 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-cl8m5"] Nov 03 20:20:40.565503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.565443 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:20:40.570860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.570747 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:40.574043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:40.574024 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-bmpqx"] Nov 03 20:20:41.064161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.053477 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:20:41.071363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.071329 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:41.071575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.071552 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:41.071815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.071683 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:41.171136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.170675 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:41.171136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.170732 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:41.191980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.191867 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:41.192255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.192230 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:41.282569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.282531 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.664227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.640689 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc68032-d375-4667-84b7-59389b5fb9ee" path="/var/lib/kubelet/pods/0bc68032-d375-4667-84b7-59389b5fb9ee/volumes" Nov 03 20:20:41.666063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.666031 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" path="/var/lib/kubelet/pods/16a41eb9-59df-42a4-81eb-4413e5b204a2/volumes" Nov 03 20:20:41.666977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.666958 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b57a418-15d9-488d-9007-9647debd6e6f" path="/var/lib/kubelet/pods/2b57a418-15d9-488d-9007-9647debd6e6f/volumes" Nov 03 20:20:41.692240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.671447 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953aca2f-55e6-4c11-b3d7-9a90a842b955" path="/var/lib/kubelet/pods/953aca2f-55e6-4c11-b3d7-9a90a842b955/volumes" Nov 03 20:20:41.693361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.693332 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ab4577-de9d-4a19-8e00-9f546754719c" path="/var/lib/kubelet/pods/b8ab4577-de9d-4a19-8e00-9f546754719c/volumes" Nov 03 20:20:41.716129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.694138 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e0b48e-8d31-489e-8759-281e75132e4e" path="/var/lib/kubelet/pods/e3e0b48e-8d31-489e-8759-281e75132e4e/volumes" Nov 03 20:20:41.728547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.728507 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerStarted","Data":"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16"} Nov 03 20:20:41.729758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:41.729733 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerStarted","Data":"0b87284a03a50d71615399a6e8aa839e087594a0cfa9a0eaadbf626164ea45eb"} Nov 03 20:20:42.106185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.102720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:42.170738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.169534 5954 patch_prober.go:29] interesting pod/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:20:42.170738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.169593 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 20:20:42.194209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.193498 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:20:42.194209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.193573 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 20:20:42.393910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.393246 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:42.400196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.400168 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:42.429128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.428819 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podStartSLOduration=4.428796943 podStartE2EDuration="4.428796943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:41.767684929 +0000 UTC m=+1382.492692527" watchObservedRunningTime="2026-11-03 20:20:42.428796943 +0000 UTC m=+1383.153804540" Nov 03 20:20:42.807283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.805722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:42.810678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.810649 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:42.811524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:42.811498 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:20:43.174518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.174488 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:20:43.174978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.174965 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:20:43.175724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.175701 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:20:43.196351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.196323 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:20:43.197924 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.197903 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:20:43.607251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.607207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:43.732475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.732439 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:43.993296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:43.993212 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.100780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.100723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.485402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.485364 5954 scope.go:122] "RemoveContainer" containerID="08a1545ae88690fa82b11a0ebd3daf363ded5df40fd4ab2dbb5403c894251a53" Nov 03 20:20:44.495933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.495910 5954 scope.go:122] "RemoveContainer" containerID="1126f829db9b23dd37b8e049fe8bfdb66a8976530598c1ed13eca87aceb72cda" Nov 03 20:20:44.508339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.508315 5954 scope.go:122] "RemoveContainer" containerID="8d54af2508fbdbdb3a5695d0578f6f056f21a43ffa44acabfc039e0e5719e05b" Nov 03 20:20:44.518332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.518309 5954 scope.go:122] "RemoveContainer" containerID="ac63e311aaccae7755e38e40d0f0df645e0a3d0ba87b2e24f493f4d87e9691f4" Nov 03 20:20:44.527992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.527973 5954 scope.go:122] "RemoveContainer" containerID="d603ecb7186a160a9ff857c169d8dd2b35e5a6d87c9672493b91c7475c9fd8ac" Nov 03 20:20:44.532796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:44.532776 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:45.100964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:45.100925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:45.211332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:45.211300 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:45.316309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:45.316274 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:45.729131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:45.728694 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:46.576420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:46.576387 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.491295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:47.491265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.951948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:47.951916 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.023755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.023723 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:20:48.023994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.023969 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" podUID="a062c118-5e7a-46dd-8399-80b3a7b3c986" containerName="maas-api" containerID="cri-o://db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d" gracePeriod=30 Nov 03 20:20:48.063473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.063064 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.295257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.268674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.631561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.618536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.724468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.724062 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:20:48.785096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.785061 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.898387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.898306 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\") pod \"a062c118-5e7a-46dd-8399-80b3a7b3c986\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " Nov 03 20:20:48.898548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.898507 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\") pod \"a062c118-5e7a-46dd-8399-80b3a7b3c986\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " Nov 03 20:20:48.898700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.898679 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\" (UniqueName: \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\") pod \"a062c118-5e7a-46dd-8399-80b3a7b3c986\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " Nov 03 20:20:48.898790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.898770 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\") pod \"a062c118-5e7a-46dd-8399-80b3a7b3c986\" (UID: \"a062c118-5e7a-46dd-8399-80b3a7b3c986\") " Nov 03 20:20:48.899267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.899237 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca" pod "a062c118-5e7a-46dd-8399-80b3a7b3c986" (UID: "a062c118-5e7a-46dd-8399-80b3a7b3c986"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:48.899887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.899758 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a062c118-5e7a-46dd-8399-80b3a7b3c986-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:48.900983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.900956 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs" pod "a062c118-5e7a-46dd-8399-80b3a7b3c986" (UID: "a062c118-5e7a-46dd-8399-80b3a7b3c986"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:48.901071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.901050 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls" pod "a062c118-5e7a-46dd-8399-80b3a7b3c986" (UID: "a062c118-5e7a-46dd-8399-80b3a7b3c986"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:48.901174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.901092 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296" pod "a062c118-5e7a-46dd-8399-80b3a7b3c986" (UID: "a062c118-5e7a-46dd-8399-80b3a7b3c986"). InnerVolumeSpecName "kube-api-access-fc296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:48.935339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.923037 5954 generic.go:356] "Generic (PLEG): container finished" podID="a062c118-5e7a-46dd-8399-80b3a7b3c986" containerID="db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d" exitCode=0 Nov 03 20:20:48.935339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.923210 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" event={"ID":"a062c118-5e7a-46dd-8399-80b3a7b3c986","Type":"ContainerDied","Data":"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d"} Nov 03 20:20:48.935339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.923264 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" event={"ID":"a062c118-5e7a-46dd-8399-80b3a7b3c986","Type":"ContainerDied","Data":"a1a6900835d8ba578998b878286627c773ab065d887997d668b1428ae85fa7be"} Nov 03 20:20:48.935339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.923297 5954 scope.go:122] "RemoveContainer" containerID="db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d" Nov 03 20:20:48.935339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.923472 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng" Nov 03 20:20:48.964683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.964071 5954 scope.go:122] "RemoveContainer" containerID="db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d" Nov 03 20:20:48.964683 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:20:48.964527 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d\": container with ID starting with db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d not found: ID does not exist" containerID="db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d" Nov 03 20:20:48.964683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.964566 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d"} err="failed to get container status \"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d\": rpc error: code = NotFound desc = could not find container \"db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d\": container with ID starting with db29bd6561ccd616d5f3a26e84a429941e9555d0a023da211b14d5934147d63d not found: ID does not exist" Nov 03 20:20:48.983838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.980916 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:20:48.988407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:48.988359 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ea43c-8fc6c9bc8-qw2ng"] Nov 03 20:20:49.014129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.005864 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:49.014129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.005904 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc296\" (UniqueName: \"kubernetes.io/projected/a062c118-5e7a-46dd-8399-80b3a7b3c986-kube-api-access-fc296\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:49.014129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.005922 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a062c118-5e7a-46dd-8399-80b3a7b3c986-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:49.071130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.069725 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.409602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.409568 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a062c118-5e7a-46dd-8399-80b3a7b3c986" path="/var/lib/kubelet/pods/a062c118-5e7a-46dd-8399-80b3a7b3c986/volumes" Nov 03 20:20:49.682729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:49.682636 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:50.485712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:50.485675 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.313409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:51.313374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.104709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:52.104666 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.776545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:52.776515 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:53.025890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:53.025857 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np"] Nov 03 20:20:53.029710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:53.029644 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8550ba63-895np"] Nov 03 20:20:53.406726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:53.406681 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec46fb5b-7dfc-4f0f-8b03-4a15197200bf" path="/var/lib/kubelet/pods/ec46fb5b-7dfc-4f0f-8b03-4a15197200bf/volumes" Nov 03 20:20:53.504345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:53.504309 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:53.818522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:53.818437 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:54.200287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:54.200252 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:54.983603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:54.983572 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:55.662417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:55.662363 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.047533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:56.026408 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.127963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:56.127895 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.391528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:56.391494 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.498517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:56.498483 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:57.172731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:57.172701 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:20:57.785477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:57.785439 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.265721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.265683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.720357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.718361 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:20:58.720870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.720575 5954 cpu_manager.go:414] "containerMap: removing container" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" Nov 03 20:20:58.720870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.720613 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" Nov 03 20:20:58.720870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.720636 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a062c118-5e7a-46dd-8399-80b3a7b3c986" containerName="maas-api" Nov 03 20:20:58.720870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.720646 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a062c118-5e7a-46dd-8399-80b3a7b3c986" containerName="maas-api" Nov 03 20:20:58.723187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.721250 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a41eb9-59df-42a4-81eb-4413e5b204a2" containerName="istio-proxy" Nov 03 20:20:58.723187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.721275 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a062c118-5e7a-46dd-8399-80b3a7b3c986" containerName="maas-api" Nov 03 20:20:58.727781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.727759 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:20:58.728098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.728006 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.740569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.740542 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-7fd1a7-openshift-default-dockercfg-pbw8k\"" Nov 03 20:20:58.741962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.741939 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.835938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.835901 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836030 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnrk\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836084 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836422 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836494 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836540 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836628 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836713 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.836918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.836894 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940350 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940494 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940557 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940592 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940644 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940782 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940816 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940869 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnrk\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.940901 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.941531 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.941542 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.941773 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.942199 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.946128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.942451 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:58.981652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.980949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.992913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:58.992668 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:59.004042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.003920 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnrk\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:59.004615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.004590 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:59.015397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.011024 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\") pod \"e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:59.105130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.096436 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:20:59.624186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.555037 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.819407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.712038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.839911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.828204 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:20:59.883908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.883432 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.922862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.922818 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:59.923078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.923056 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:59.923252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.923231 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:20:59.959567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.959530 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:20:59.959736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.959585 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:20:59.959736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.959691 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:20:59.964489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:20:59.964462 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2d5927-serving-cert\"" Nov 03 20:21:00.060155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.060098 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.078129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.060451 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.078129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.060547 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqk5\" (UniqueName: \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.078129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.060735 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.186645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.185512 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.186645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.185623 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.186645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.185732 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqk5\" (UniqueName: \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.186645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.185953 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.223130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.221360 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.247132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.240962 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerStarted","Data":"68820c43ba536802ef65069bd34363312483ab6584d18e43c6bbcf143e3428ff"} Nov 03 20:21:00.255131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.254802 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.278130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.255358 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.303454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.303421 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqk5\" (UniqueName: \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\") pod \"maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.462893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:00.462616 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:00.658210 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:00.641821 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:21:01.340518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.340422 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.352334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.352290 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:21:01.386130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.353397 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:21:01.605258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.341173 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.605258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.446287 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.605258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.505935 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.605258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.522466 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.605258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.575227 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.629764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.629722 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-64x5n" podUID="3602a4e5-7b74-44e0-a640-3e608c97bf19" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.964189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:01.948483 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podStartSLOduration=3.948458272 podStartE2EDuration="3.948458272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:01.881682398 +0000 UTC m=+1402.606689995" watchObservedRunningTime="2026-11-03 20:21:01.948458272 +0000 UTC m=+1402.673465870" Nov 03 20:21:02.039159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.015802 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerStarted","Data":"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9"} Nov 03 20:21:02.039159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.015965 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:21:02.041695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.041555 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:02.152134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.150767 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:21:02.164774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.164747 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:21:02.214313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.207464 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:21:02.429130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.427463 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.445131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.432544 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:21:02.445131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.432600 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:21:02.445131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.432873 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" containerID="cri-o://895250e8b10996f794666540583eb6a6e7a8de7dc95c6eb7b6379c2d22c44836" gracePeriod=30 Nov 03 20:21:02.492648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.492471 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 20:21:02.522897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.522817 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:21:02.523465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.523435 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" containerID="cri-o://3f75dc601680c479b1d7013923f8a107819a4fdfcc70062013447133b30cf1a6" gracePeriod=30 Nov 03 20:21:02.566483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.566425 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.567952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.567926 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:21:02.587255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.581422 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.591464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.587763 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:21:02.592915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.592501 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.594507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.593157 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:21:02.639253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:02.629951 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:21:03.228054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.227640 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:21:03.228054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.228041 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" containerID="cri-o://2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4" gracePeriod=30 Nov 03 20:21:03.230128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.229294 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:21:03.256100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.255587 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:21:03.256100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.255625 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" event={"ID":"8036b665-c559-434d-bb41-2e040e3ce501","Type":"ContainerStarted","Data":"707e8399e42a7884d64f97d0518d3c0d8531a7027d4328eeb5c37308feadab8c"} Nov 03 20:21:03.256100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.255665 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:21:03.260344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.260154 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:21:03.366179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.349454 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:21:03.609644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:03.609608 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:04.264638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:04.264592 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" event={"ID":"8036b665-c559-434d-bb41-2e040e3ce501","Type":"ContainerStarted","Data":"135de9e3842b1ea71b1fec568f91e43742691a1b7a03a71fa4f8d9144924ef3b"} Nov 03 20:21:04.265077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:04.264723 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:04.288477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:04.288430 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" podStartSLOduration=4.261127988 podStartE2EDuration="5.288411676s" totalImagesPullingTime="1.027283688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:04.286608881 +0000 UTC m=+1405.011616477" watchObservedRunningTime="2026-11-03 20:21:04.288411676 +0000 UTC m=+1405.013419271" Nov 03 20:21:04.344944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:04.344907 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:04.581227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:04.581076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.553269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:05.553234 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.671416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:05.671384 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:06.666541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:06.666509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:07.085563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:07.085453 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:07.279629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:07.279582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:07.720406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:07.720367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:08.336344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.336168 5954 generic.go:356] "Generic (PLEG): container finished" podID="3c02de98-e73f-4542-b221-d088eff28d0d" containerID="895250e8b10996f794666540583eb6a6e7a8de7dc95c6eb7b6379c2d22c44836" exitCode=0 Nov 03 20:21:08.336344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.336239 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" event={"ID":"3c02de98-e73f-4542-b221-d088eff28d0d","Type":"ContainerDied","Data":"895250e8b10996f794666540583eb6a6e7a8de7dc95c6eb7b6379c2d22c44836"} Nov 03 20:21:08.338385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.338359 5954 generic.go:356] "Generic (PLEG): container finished" podID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerID="3f75dc601680c479b1d7013923f8a107819a4fdfcc70062013447133b30cf1a6" exitCode=0 Nov 03 20:21:08.338509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.338494 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" event={"ID":"2f978df8-2422-4339-a4f0-6dd30f39b9c0","Type":"ContainerDied","Data":"3f75dc601680c479b1d7013923f8a107819a4fdfcc70062013447133b30cf1a6"} Nov 03 20:21:08.432336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.432301 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:21:08.491522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.491444 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.491522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.491498 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.491758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.491571 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.491822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.491755 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.492363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.492332 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.492486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.492384 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.492486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.492452 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.492611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.492489 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.492611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.492518 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\") pod \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\" (UID: \"2f978df8-2422-4339-a4f0-6dd30f39b9c0\") " Nov 03 20:21:08.493917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.493362 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.493917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.493643 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.494084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.493941 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.495140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.494321 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:08.495140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.494586 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.495140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.494902 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.498064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.498038 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.498249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.498076 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "kube-api-access-w62s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.499579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.499559 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.500382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.500348 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo" pod "2f978df8-2422-4339-a4f0-6dd30f39b9c0" (UID: "2f978df8-2422-4339-a4f0-6dd30f39b9c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:08.506985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.506918 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:21:08.572334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.572294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:08.585231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.585204 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:21:08.594419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594397 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.594532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594450 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.594532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594479 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.594532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594525 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.594726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594607 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.594726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594640 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.594726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594718 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.594858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594782 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.594858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594823 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.594922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594889 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.594967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.594950 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.595041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595026 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.595095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595035 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.595095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595056 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.595095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595061 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.595270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595143 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.595324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595290 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:08.595379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595348 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.595555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595536 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.595673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595579 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.595673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595602 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.595673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595610 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.595673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595613 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\") pod \"3c02de98-e73f-4542-b221-d088eff28d0d\" (UID: \"3c02de98-e73f-4542-b221-d088eff28d0d\") " Nov 03 20:21:08.595894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595705 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\") pod \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\" (UID: \"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b\") " Nov 03 20:21:08.595894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.595813 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.596159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596141 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.596245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596179 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:08.596491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596447 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596499 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596522 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596537 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596550 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62s8\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-kube-api-access-w62s8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596563 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596577 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596589 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596605 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c02de98-e73f-4542-b221-d088eff28d0d-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596620 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596633 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596646 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596669 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596685 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596699 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596714 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596728 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.596987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.596742 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f978df8-2422-4339-a4f0-6dd30f39b9c0-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.598028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598001 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.598292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598229 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:08.598635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598583 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.598736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598695 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "kube-api-access-fp4v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.598736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598696 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:08.598890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.598867 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.599144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.599081 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq" pod "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" (UID: "c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b"). InnerVolumeSpecName "kube-api-access-qmkgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:08.600352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.600333 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy" pod "3c02de98-e73f-4542-b221-d088eff28d0d" (UID: "3c02de98-e73f-4542-b221-d088eff28d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:08.698862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698828 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.698862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698858 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.698862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698868 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698878 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmkgq\" (UniqueName: \"kubernetes.io/projected/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b-kube-api-access-qmkgq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698887 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698896 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c02de98-e73f-4542-b221-d088eff28d0d-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698906 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698915 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4v5\" (UniqueName: \"kubernetes.io/projected/3c02de98-e73f-4542-b221-d088eff28d0d-kube-api-access-fp4v5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:08.699066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:08.698930 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c02de98-e73f-4542-b221-d088eff28d0d-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:09.349420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.349387 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" Nov 03 20:21:09.349420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.349408 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w" event={"ID":"3c02de98-e73f-4542-b221-d088eff28d0d","Type":"ContainerDied","Data":"40e03e7945e96f62631bb62e2cbb1f83034833d83c98a7fd9b2026f5a7c30dc0"} Nov 03 20:21:09.349932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.349475 5954 scope.go:122] "RemoveContainer" containerID="895250e8b10996f794666540583eb6a6e7a8de7dc95c6eb7b6379c2d22c44836" Nov 03 20:21:09.350959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.350939 5954 generic.go:356] "Generic (PLEG): container finished" podID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerID="2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4" exitCode=0 Nov 03 20:21:09.351053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.351038 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" event={"ID":"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b","Type":"ContainerDied","Data":"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4"} Nov 03 20:21:09.351127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.351077 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" event={"ID":"c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b","Type":"ContainerDied","Data":"bb9055a0a40280338d90f3458d8c9a5d61bdf195548159d513848faf8d8ed11c"} Nov 03 20:21:09.351189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.351175 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf" Nov 03 20:21:09.357367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.357242 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" event={"ID":"2f978df8-2422-4339-a4f0-6dd30f39b9c0","Type":"ContainerDied","Data":"31922f686cf4339521c121560840b4a439224b51817d3193f6f9986684b58102"} Nov 03 20:21:09.357367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.357303 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv" Nov 03 20:21:09.365557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.365521 5954 scope.go:122] "RemoveContainer" containerID="2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4" Nov 03 20:21:09.382591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.382572 5954 scope.go:122] "RemoveContainer" containerID="2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4" Nov 03 20:21:09.383860 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:09.383476 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4\": container with ID starting with 2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4 not found: ID does not exist" containerID="2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4" Nov 03 20:21:09.383860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.383526 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4"} err="failed to get container status \"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4\": rpc error: code = NotFound desc = could not find container \"2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4\": container with ID starting with 2a6ec159c0917171d47d025c626a96908435fde0b02549566f65688b0af431e4 not found: ID does not exist" Nov 03 20:21:09.383860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.383551 5954 scope.go:122] "RemoveContainer" containerID="3f75dc601680c479b1d7013923f8a107819a4fdfcc70062013447133b30cf1a6" Nov 03 20:21:09.386325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.386305 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:21:09.390938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.390912 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-j8b7w"] Nov 03 20:21:09.401081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.401060 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:09.411898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.411874 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" path="/var/lib/kubelet/pods/3c02de98-e73f-4542-b221-d088eff28d0d/volumes" Nov 03 20:21:09.412227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.412212 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:21:09.412277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.412239 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bqhsf"] Nov 03 20:21:09.424593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.424565 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:21:09.429999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:09.429981 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-v7fcv"] Nov 03 20:21:10.141371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:10.141325 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:10.291378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:10.291347 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:21:10.988872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:10.988837 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:11.406886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:11.406851 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" path="/var/lib/kubelet/pods/2f978df8-2422-4339-a4f0-6dd30f39b9c0/volumes" Nov 03 20:21:11.407430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:11.407415 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" path="/var/lib/kubelet/pods/c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b/volumes" Nov 03 20:21:12.290249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:12.290204 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:13.326651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:13.326550 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:14.089051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:14.089006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:14.770371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:14.770334 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:15.691495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:15.691460 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:16.482781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:16.482747 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:17.178171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.178136 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:21:17.178391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.178367 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" containerID="cri-o://8e166213a1df294ababda9df37a669b509ab7ad411a9ea38a1d8f9256768c7ef" gracePeriod=30 Nov 03 20:21:17.183698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.183673 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:21:17.183943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.183896 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" containerID="cri-o://2d91796ec428550701beab6fbd3059670d1f5a3b497d02cf8dc5fa841c2cc58e" gracePeriod=30 Nov 03 20:21:17.188669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.188638 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:21:17.188881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.188858 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" containerID="cri-o://bf1dfad88173ffec6d0e94577ef1616b010f1fc5ab7d814cf559b52320a4f0f7" gracePeriod=30 Nov 03 20:21:17.199676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.199650 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:17.240743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.240704 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:21:17.242468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.242332 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" containerID="cri-o://abce18e69927634c2e3f0614986ba46e0ef3af29e7d9814676dac45706619240" gracePeriod=30 Nov 03 20:21:17.246120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.246078 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:21:17.246303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.246279 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" containerID="cri-o://097478e56839e01a3a0024719943b2d24761c4cd056c3590c427231b2082d19d" gracePeriod=30 Nov 03 20:21:17.395737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.395667 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:21:17.395737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:17.395732 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:21:18.026177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:18.023249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.019563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.019268 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:21:19.020352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020328 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" Nov 03 20:21:19.020352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020349 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020365 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020370 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020418 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020423 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020531 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c02de98-e73f-4542-b221-d088eff28d0d" containerName="istio-proxy" Nov 03 20:21:19.020543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020545 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fb06eb-ee08-4fd4-8bd9-8eca9fdd4d2b" containerName="istio-proxy" Nov 03 20:21:19.020863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.020558 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f978df8-2422-4339-a4f0-6dd30f39b9c0" containerName="istio-proxy" Nov 03 20:21:19.026563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.026541 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.040490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.040448 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:21:19.061332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.061306 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-7fd1a7-serving-cert\"" Nov 03 20:21:19.080476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.080445 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.097964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.097940 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.098095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.098071 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bb8s\" (UniqueName: \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.098191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.098171 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.098286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.098266 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.201146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.200774 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.201146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.200904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bb8s\" (UniqueName: \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.201146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.200959 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.201146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.201054 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.202248 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:19.202220 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-serving-cert: secret "maas-api-e2e-shared-a-w1-7fd1a7-serving-cert" not found Nov 03 20:21:19.202392 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:19.202307 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls podName:6b3a20fd-5864-4a72-88ea-b7a74b6f41d7 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:19.702285772 +0000 UTC m=+1420.427293360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls") pod "maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" (UID: "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7") : secret "maas-api-e2e-shared-a-w1-7fd1a7-serving-cert" not found Nov 03 20:21:19.236129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.203252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.236129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.217290 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.237962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.237928 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bb8s\" (UniqueName: \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.239905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.239878 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.471002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.470964 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.573362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.573329 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.739219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.728190 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.763875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.760843 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:19.959205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:19.950596 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:20.355511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.333911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:20.412747 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:20.412707 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3a20fd_5864_4a72_88ea_b7a74b6f41d7.slice/crio-b8607392ad6585d8e2651796ff7476d239a922a4a82eb628f7b3058a2265a8ac WatchSource:0}: Error finding container b8607392ad6585d8e2651796ff7476d239a922a4a82eb628f7b3058a2265a8ac: Status 404 returned error can't find the container with id b8607392ad6585d8e2651796ff7476d239a922a4a82eb628f7b3058a2265a8ac Nov 03 20:21:20.415100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.414912 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:21:20.445891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.445853 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:20.624544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.624454 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" event={"ID":"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7","Type":"ContainerStarted","Data":"b8607392ad6585d8e2651796ff7476d239a922a4a82eb628f7b3058a2265a8ac"} Nov 03 20:21:20.892205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.890576 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:21:20.903489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.903142 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:21:20.903489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.903210 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:20.912976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.912951 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e50c04-openshift-default-dockercfg-24p6q\"" Nov 03 20:21:20.937246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:20.937216 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026020 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026133 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026206 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026270 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026441 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026522 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026575 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026602 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.027051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.026716 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnbd\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.126645 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128078 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128157 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128184 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128289 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnbd\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128375 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128431 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128508 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128569 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.134191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.128767 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.150932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.150848 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.151548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.151519 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.159567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.159536 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.160005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.159983 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.179125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.174930 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.320001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.315624 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.320560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.320491 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.359761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.331534 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.376249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.365222 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnbd\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\") pod \"e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.569444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.569166 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:21.646010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.644699 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" event={"ID":"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7","Type":"ContainerStarted","Data":"382f47927eba6c50a45f330807e0226d47a25318bc4e5a2baaa21ea7b1860bfd"} Nov 03 20:21:21.646010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.645939 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:21.669130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.669036 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" podStartSLOduration=3.172790066 podStartE2EDuration="3.669019847s" totalImagesPullingTime="496.229781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:21.666933642 +0000 UTC m=+1422.391941238" watchObservedRunningTime="2026-11-03 20:21:21.669019847 +0000 UTC m=+1422.394027443" Nov 03 20:21:21.691601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.691572 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.789856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.789797 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.983130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:21.982336 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:22.112055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.112018 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:21:22.243181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.243005 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:22.249815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.243128 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:22.249815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.249680 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:22.684281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.684116 5954 generic.go:356] "Generic (PLEG): container finished" podID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerID="8e166213a1df294ababda9df37a669b509ab7ad411a9ea38a1d8f9256768c7ef" exitCode=0 Nov 03 20:21:22.684281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.684216 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" event={"ID":"b97a5867-0ca8-4d69-988e-fedf672af29b","Type":"ContainerDied","Data":"8e166213a1df294ababda9df37a669b509ab7ad411a9ea38a1d8f9256768c7ef"} Nov 03 20:21:22.689092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.688938 5954 generic.go:356] "Generic (PLEG): container finished" podID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerID="2d91796ec428550701beab6fbd3059670d1f5a3b497d02cf8dc5fa841c2cc58e" exitCode=0 Nov 03 20:21:22.689092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.689018 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" event={"ID":"46f7548a-59dd-4961-aa1f-94a4789b890a","Type":"ContainerDied","Data":"2d91796ec428550701beab6fbd3059670d1f5a3b497d02cf8dc5fa841c2cc58e"} Nov 03 20:21:22.692427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.692399 5954 generic.go:356] "Generic (PLEG): container finished" podID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerID="abce18e69927634c2e3f0614986ba46e0ef3af29e7d9814676dac45706619240" exitCode=0 Nov 03 20:21:22.692882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.692592 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" event={"ID":"bd479ced-6534-4776-a30f-c0e56231a0a8","Type":"ContainerDied","Data":"abce18e69927634c2e3f0614986ba46e0ef3af29e7d9814676dac45706619240"} Nov 03 20:21:22.694900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.694882 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" event={"ID":"21debdd0-6616-40ec-8610-e5a6c3149fcc","Type":"ContainerDied","Data":"097478e56839e01a3a0024719943b2d24761c4cd056c3590c427231b2082d19d"} Nov 03 20:21:22.695066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.694758 5954 generic.go:356] "Generic (PLEG): container finished" podID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerID="097478e56839e01a3a0024719943b2d24761c4cd056c3590c427231b2082d19d" exitCode=0 Nov 03 20:21:22.697399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.697164 5954 generic.go:356] "Generic (PLEG): container finished" podID="cadcc3b3-377d-4478-b91c-c2957577306a" containerID="bf1dfad88173ffec6d0e94577ef1616b010f1fc5ab7d814cf559b52320a4f0f7" exitCode=0 Nov 03 20:21:22.697399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.697339 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" event={"ID":"cadcc3b3-377d-4478-b91c-c2957577306a","Type":"ContainerDied","Data":"bf1dfad88173ffec6d0e94577ef1616b010f1fc5ab7d814cf559b52320a4f0f7"} Nov 03 20:21:22.701472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.701451 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerStarted","Data":"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e"} Nov 03 20:21:22.701762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.701746 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerStarted","Data":"222e9b2a10c52e1a2ea2147af55d490982f12f30b399099181c0517da75fbf96"} Nov 03 20:21:22.747856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.747795 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podStartSLOduration=2.747775582 podStartE2EDuration="2.747775582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:22.729220241 +0000 UTC m=+1423.454227836" watchObservedRunningTime="2026-11-03 20:21:22.747775582 +0000 UTC m=+1423.472783180" Nov 03 20:21:22.945347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.945307 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:21:22.948697 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:22.948656 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:21:23.108042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.107988 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.108272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108100 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.108358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108287 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.108358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108345 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.108472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108366 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.108472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108403 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.108472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108457 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.108639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108587 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.108704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108652 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.108704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108655 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.108819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108717 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.108969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.108931 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.109053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109036 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.109146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109086 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.109227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109167 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.109287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109218 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.109287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109265 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.109693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109422 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.109693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109442 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.109693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109489 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\") pod \"b97a5867-0ca8-4d69-988e-fedf672af29b\" (UID: \"b97a5867-0ca8-4d69-988e-fedf672af29b\") " Nov 03 20:21:23.109693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109536 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.109693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.109594 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\") pod \"46f7548a-59dd-4961-aa1f-94a4789b890a\" (UID: \"46f7548a-59dd-4961-aa1f-94a4789b890a\") " Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.110493 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.110522 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.110538 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.110554 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.110749 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.111144 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.111555 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.112013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.111766 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.112857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.112751 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.116390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.116353 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.117043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.116741 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.119234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.119169 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.138055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.137995 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.152177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.150325 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.152177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.150439 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.152177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.150476 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "kube-api-access-77nnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.152177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.150517 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k" pod "46f7548a-59dd-4961-aa1f-94a4789b890a" (UID: "46f7548a-59dd-4961-aa1f-94a4789b890a"). InnerVolumeSpecName "kube-api-access-2tg4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.152177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.151954 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:23.167725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.167685 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert" pod "b97a5867-0ca8-4d69-988e-fedf672af29b" (UID: "b97a5867-0ca8-4d69-988e-fedf672af29b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213876 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213912 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213928 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nnx\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-kube-api-access-77nnx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213943 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213959 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97a5867-0ca8-4d69-988e-fedf672af29b-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213974 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.213989 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214003 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97a5867-0ca8-4d69-988e-fedf672af29b-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214017 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46f7548a-59dd-4961-aa1f-94a4789b890a-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214030 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97a5867-0ca8-4d69-988e-fedf672af29b-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214046 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214060 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg4k\" (UniqueName: \"kubernetes.io/projected/46f7548a-59dd-4961-aa1f-94a4789b890a-kube-api-access-2tg4k\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214075 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.250669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.214090 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46f7548a-59dd-4961-aa1f-94a4789b890a-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.332980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.331714 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:21:23.336972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.334365 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:21:23.336972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.336473 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432201 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432258 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432284 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432316 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432346 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432379 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432404 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432451 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432495 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432589 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432641 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432721 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432752 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432788 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432830 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432886 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432949 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.432972 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433021 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.433046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433048 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433084 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433140 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433187 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\") pod \"cadcc3b3-377d-4478-b91c-c2957577306a\" (UID: \"cadcc3b3-377d-4478-b91c-c2957577306a\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433230 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433323 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433414 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\") pod \"21debdd0-6616-40ec-8610-e5a6c3149fcc\" (UID: \"21debdd0-6616-40ec-8610-e5a6c3149fcc\") " Nov 03 20:21:23.434368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.433460 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\") pod \"bd479ced-6534-4776-a30f-c0e56231a0a8\" (UID: \"bd479ced-6534-4776-a30f-c0e56231a0a8\") " Nov 03 20:21:23.437762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.437727 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.437946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.437925 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.438169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.438148 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.438798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.438766 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.441859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.440858 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.441859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.441065 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.441859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.441369 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.441859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.441575 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.443100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.443079 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.443723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.443680 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.443937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.443916 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.454812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.454362 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.454812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.454444 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.454812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.454464 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.454812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.454596 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.454812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.454714 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.463344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.463266 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.463469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.463340 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.463723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.463703 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.466824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.466792 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.467778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.467755 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.467873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.467772 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "kube-api-access-mwgpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.467873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.467813 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token" pod "bd479ced-6534-4776-a30f-c0e56231a0a8" (UID: "bd479ced-6534-4776-a30f-c0e56231a0a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.468503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.468475 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.468906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.468881 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "kube-api-access-zw2zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.469069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.469049 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy" pod "cadcc3b3-377d-4478-b91c-c2957577306a" (UID: "cadcc3b3-377d-4478-b91c-c2957577306a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.470291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.470272 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5" pod "21debdd0-6616-40ec-8610-e5a6c3149fcc" (UID: "21debdd0-6616-40ec-8610-e5a6c3149fcc"). InnerVolumeSpecName "kube-api-access-vmnn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.541486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541449 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadcc3b3-377d-4478-b91c-c2957577306a-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541484 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541499 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541513 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541526 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541539 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541550 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541562 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541574 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541586 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541598 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnn5\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-kube-api-access-vmnn5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541611 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwgpt\" (UniqueName: \"kubernetes.io/projected/bd479ced-6534-4776-a30f-c0e56231a0a8-kube-api-access-mwgpt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541624 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541637 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541649 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541661 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21debdd0-6616-40ec-8610-e5a6c3149fcc-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541673 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadcc3b3-377d-4478-b91c-c2957577306a-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541684 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw2zc\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-kube-api-access-zw2zc\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541697 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541711 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541725 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd479ced-6534-4776-a30f-c0e56231a0a8-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541742 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21debdd0-6616-40ec-8610-e5a6c3149fcc-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.541752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541757 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd479ced-6534-4776-a30f-c0e56231a0a8-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.542793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541770 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadcc3b3-377d-4478-b91c-c2957577306a-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.542793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541785 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd479ced-6534-4776-a30f-c0e56231a0a8-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.542793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541799 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21debdd0-6616-40ec-8610-e5a6c3149fcc-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.542793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.541812 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadcc3b3-377d-4478-b91c-c2957577306a-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.575264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.575226 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:23.581762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.581734 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:23.712503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.712427 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" Nov 03 20:21:23.712910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.712429 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8" event={"ID":"21debdd0-6616-40ec-8610-e5a6c3149fcc","Type":"ContainerDied","Data":"ddb532404ad3c886460ed5d29bb02c20530ab9e9dde14227809d1ce08fe8f887"} Nov 03 20:21:23.712910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.712561 5954 scope.go:122] "RemoveContainer" containerID="097478e56839e01a3a0024719943b2d24761c4cd056c3590c427231b2082d19d" Nov 03 20:21:23.714395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.714285 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" Nov 03 20:21:23.714395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.714320 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw" event={"ID":"cadcc3b3-377d-4478-b91c-c2957577306a","Type":"ContainerDied","Data":"197a36a20d0b1bff3892481a674c7e0af77f667adc71e335e0a829166151f24e"} Nov 03 20:21:23.716799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.716778 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" Nov 03 20:21:23.716966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.716807 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5" event={"ID":"b97a5867-0ca8-4d69-988e-fedf672af29b","Type":"ContainerDied","Data":"c9f0a3ee03d9985d6a974b5b85d3000e207004780473c01d3e8e016b365b083a"} Nov 03 20:21:23.718844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.718820 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" event={"ID":"46f7548a-59dd-4961-aa1f-94a4789b890a","Type":"ContainerDied","Data":"b916367e3c53c0a3d9dbc33695c05eedcb7193e439f994ee270b7fd465b5f2d1"} Nov 03 20:21:23.718844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.718829 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns" Nov 03 20:21:23.720826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.720630 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" Nov 03 20:21:23.720979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.720673 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4" event={"ID":"bd479ced-6534-4776-a30f-c0e56231a0a8","Type":"ContainerDied","Data":"62b2dc22e8306c1ffcd1435f64cb0bd3eab74cfd5bb1adfa5420b51cde3e771c"} Nov 03 20:21:23.725530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.725506 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:23.726284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.726259 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:21:23.730640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.730545 5954 scope.go:122] "RemoveContainer" containerID="bf1dfad88173ffec6d0e94577ef1616b010f1fc5ab7d814cf559b52320a4f0f7" Nov 03 20:21:23.744685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.744555 5954 scope.go:122] "RemoveContainer" containerID="8e166213a1df294ababda9df37a669b509ab7ad411a9ea38a1d8f9256768c7ef" Nov 03 20:21:23.752733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.752675 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:21:23.757953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.757923 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-w7qw8"] Nov 03 20:21:23.765370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.765342 5954 scope.go:122] "RemoveContainer" containerID="2d91796ec428550701beab6fbd3059670d1f5a3b497d02cf8dc5fa841c2cc58e" Nov 03 20:21:23.783248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.783220 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:21:23.806661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.802717 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-jvwdw"] Nov 03 20:21:23.808040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.808009 5954 scope.go:122] "RemoveContainer" containerID="abce18e69927634c2e3f0614986ba46e0ef3af29e7d9814676dac45706619240" Nov 03 20:21:23.831199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.831164 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:21:23.847594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.847567 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-d5bn4"] Nov 03 20:21:23.849972 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:23.849938 5954 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd479ced_6534_4776_a30f_c0e56231a0a8.slice/crio-62b2dc22e8306c1ffcd1435f64cb0bd3eab74cfd5bb1adfa5420b51cde3e771c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd479ced_6534_4776_a30f_c0e56231a0a8.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:21:23.858304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.858255 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:21:23.865688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.865661 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-v72j5"] Nov 03 20:21:23.870259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.870235 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:21:23.874101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:23.874075 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-8hwns"] Nov 03 20:21:24.169245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:24.169209 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:24.861144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:24.861091 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:21:24.861640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:24.861384 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" containerID="cri-o://9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf" gracePeriod=30 Nov 03 20:21:24.913367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:24.913323 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.045866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.045828 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.206133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.198702 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.281876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.281817 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:21:25.475435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.436295 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\") pod \"e183ed98-afd7-473d-b143-5c5c2916ec2e\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " Nov 03 20:21:25.475946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.475886 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\") pod \"e183ed98-afd7-473d-b143-5c5c2916ec2e\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " Nov 03 20:21:25.476123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.476086 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\") pod \"e183ed98-afd7-473d-b143-5c5c2916ec2e\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " Nov 03 20:21:25.476359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.476283 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\" (UniqueName: \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\") pod \"e183ed98-afd7-473d-b143-5c5c2916ec2e\" (UID: \"e183ed98-afd7-473d-b143-5c5c2916ec2e\") " Nov 03 20:21:25.483811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.483512 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca" pod "e183ed98-afd7-473d-b143-5c5c2916ec2e" (UID: "e183ed98-afd7-473d-b143-5c5c2916ec2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:25.524622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.524588 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw" pod "e183ed98-afd7-473d-b143-5c5c2916ec2e" (UID: "e183ed98-afd7-473d-b143-5c5c2916ec2e"). InnerVolumeSpecName "kube-api-access-b5vhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:25.524816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.524644 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" path="/var/lib/kubelet/pods/21debdd0-6616-40ec-8610-e5a6c3149fcc/volumes" Nov 03 20:21:25.525741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.525608 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" path="/var/lib/kubelet/pods/46f7548a-59dd-4961-aa1f-94a4789b890a/volumes" Nov 03 20:21:25.526494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.526423 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" path="/var/lib/kubelet/pods/b97a5867-0ca8-4d69-988e-fedf672af29b/volumes" Nov 03 20:21:25.527329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.527291 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs" pod "e183ed98-afd7-473d-b143-5c5c2916ec2e" (UID: "e183ed98-afd7-473d-b143-5c5c2916ec2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:25.527954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.527781 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" path="/var/lib/kubelet/pods/bd479ced-6534-4776-a30f-c0e56231a0a8/volumes" Nov 03 20:21:25.528161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.528136 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls" pod "e183ed98-afd7-473d-b143-5c5c2916ec2e" (UID: "e183ed98-afd7-473d-b143-5c5c2916ec2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:25.528490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.528470 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" path="/var/lib/kubelet/pods/cadcc3b3-377d-4478-b91c-c2957577306a/volumes" Nov 03 20:21:25.531923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.531857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.579060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.579021 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:25.579060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.579053 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e183ed98-afd7-473d-b143-5c5c2916ec2e-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:25.579283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.579067 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e183ed98-afd7-473d-b143-5c5c2916ec2e-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:25.579283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.579081 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5vhw\" (UniqueName: \"kubernetes.io/projected/e183ed98-afd7-473d-b143-5c5c2916ec2e-kube-api-access-b5vhw\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:25.742607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.742521 5954 generic.go:356] "Generic (PLEG): container finished" podID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerID="9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf" exitCode=0 Nov 03 20:21:25.742607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.742575 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" Nov 03 20:21:25.742801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.742606 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerDied","Data":"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf"} Nov 03 20:21:25.742801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.742665 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll" event={"ID":"e183ed98-afd7-473d-b143-5c5c2916ec2e","Type":"ContainerDied","Data":"66ea018a5c7d24f8fa49616c1050c75127c9d25fd404de4340ed6399c4c33f5f"} Nov 03 20:21:25.742801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.742690 5954 scope.go:122] "RemoveContainer" containerID="9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf" Nov 03 20:21:25.752066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.752043 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.753567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.753552 5954 scope.go:122] "RemoveContainer" containerID="ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99" Nov 03 20:21:25.763614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.763589 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:21:25.765294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.765275 5954 scope.go:122] "RemoveContainer" containerID="9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf" Nov 03 20:21:25.765578 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:25.765561 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf\": container with ID starting with 9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf not found: ID does not exist" containerID="9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf" Nov 03 20:21:25.765633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.765590 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf"} err="failed to get container status \"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf\": rpc error: code = NotFound desc = could not find container \"9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf\": container with ID starting with 9012dbcf907d822d43eb0626f469c2a70e01102eb04b8a3d7b30c14ba6b688bf not found: ID does not exist" Nov 03 20:21:25.765633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.765607 5954 scope.go:122] "RemoveContainer" containerID="ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99" Nov 03 20:21:25.765912 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:25.765887 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99\": container with ID starting with ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99 not found: ID does not exist" containerID="ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99" Nov 03 20:21:25.765984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.765924 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99"} err="failed to get container status \"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99\": rpc error: code = NotFound desc = could not find container \"ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99\": container with ID starting with ef83bf04e93fb1a692b7c27b4f4c26642e62b85caa549a27babd4db6e69a5d99 not found: ID does not exist" Nov 03 20:21:25.767607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.767588 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9af0cd-85bd6b555c-l26ll"] Nov 03 20:21:25.853811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:25.853768 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.245059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.245019 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.321624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.321592 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:21:26.322400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322380 5954 cpu_manager.go:414] "containerMap: removing container" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" Nov 03 20:21:26.322400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322401 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322415 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322423 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322441 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322448 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322470 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322477 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322509 5954 cpu_manager.go:414] "containerMap: removing container" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322516 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322540 5954 cpu_manager.go:414] "containerMap: removing container" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" Nov 03 20:21:26.322586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322547 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322726 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadcc3b3-377d-4478-b91c-c2957577306a" containerName="istio-proxy" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322741 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322757 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f7548a-59dd-4961-aa1f-94a4789b890a" containerName="istio-proxy" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322767 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97a5867-0ca8-4d69-988e-fedf672af29b" containerName="istio-proxy" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322779 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd479ced-6534-4776-a30f-c0e56231a0a8" containerName="istio-proxy" Nov 03 20:21:26.323056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.322796 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="21debdd0-6616-40ec-8610-e5a6c3149fcc" containerName="istio-proxy" Nov 03 20:21:26.328086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.328066 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.333451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.333418 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e50c04\"" Nov 03 20:21:26.333762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.333743 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e50c04-dockercfg-v4wtd\"" Nov 03 20:21:26.335613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.335595 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:21:26.355316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.355282 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.388036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.388005 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:21:26.388844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.388823 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.388844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.388843 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.389071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.389058 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" containerName="maas-api" Nov 03 20:21:26.392837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.392817 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.401464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.401432 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7gz\" (UniqueName: \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.401565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.401480 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.403395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.403373 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:21:26.584611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.577716 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rbd\" (UniqueName: \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.584611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.577769 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.584611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.577886 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7gz\" (UniqueName: \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.584611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.577918 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.584611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.580042 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.656692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.656604 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7gz\" (UniqueName: \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\") pod \"payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:26.681639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.680592 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rbd\" (UniqueName: \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.681639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.680643 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.683297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.682925 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.805149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.782443 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rbd\" (UniqueName: \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\") pod \"payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.805149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.795690 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:26.935474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.935439 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.949568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:26.949538 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:27.146175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.145945 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:21:27.148399 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:27.148367 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aad01dc_a017_43fd_8c23_b328055098a5.slice/crio-171b51e57f0df6b6b2f1405ff1428f22c7ed0e8e15af8467054ceab9d06ca340 WatchSource:0}: Error finding container 171b51e57f0df6b6b2f1405ff1428f22c7ed0e8e15af8467054ceab9d06ca340: Status 404 returned error can't find the container with id 171b51e57f0df6b6b2f1405ff1428f22c7ed0e8e15af8467054ceab9d06ca340 Nov 03 20:21:27.362096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.362062 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:21:27.362453 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:27.362176 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d1eacb_df49_4907_aec0_4468bf287c0a.slice/crio-cae091c271c9713b9c02a8034653bd34a3d36035635372d87fd3f6fe0621a2af WatchSource:0}: Error finding container cae091c271c9713b9c02a8034653bd34a3d36035635372d87fd3f6fe0621a2af: Status 404 returned error can't find the container with id cae091c271c9713b9c02a8034653bd34a3d36035635372d87fd3f6fe0621a2af Nov 03 20:21:27.408347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.408319 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e183ed98-afd7-473d-b143-5c5c2916ec2e" path="/var/lib/kubelet/pods/e183ed98-afd7-473d-b143-5c5c2916ec2e/volumes" Nov 03 20:21:27.810549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.810470 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:27.918332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.918291 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" event={"ID":"73d1eacb-df49-4907-aec0-4468bf287c0a","Type":"ContainerStarted","Data":"dfa07f226cb38929d12365eb07215d837d6c993ef202fa1ecb46ce6136fcbf84"} Nov 03 20:21:27.918667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.918640 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" event={"ID":"73d1eacb-df49-4907-aec0-4468bf287c0a","Type":"ContainerStarted","Data":"cae091c271c9713b9c02a8034653bd34a3d36035635372d87fd3f6fe0621a2af"} Nov 03 20:21:27.920187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.920121 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:27.945184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.945140 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" event={"ID":"9aad01dc-a017-43fd-8c23-b328055098a5","Type":"ContainerStarted","Data":"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130"} Nov 03 20:21:27.948296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.948263 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" event={"ID":"9aad01dc-a017-43fd-8c23-b328055098a5","Type":"ContainerStarted","Data":"171b51e57f0df6b6b2f1405ff1428f22c7ed0e8e15af8467054ceab9d06ca340"} Nov 03 20:21:27.948522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.948506 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:27.967609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.967550 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" podStartSLOduration=1.9675288069999999 podStartE2EDuration="1.967528807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:27.943614831 +0000 UTC m=+1428.668622428" watchObservedRunningTime="2026-11-03 20:21:27.967528807 +0000 UTC m=+1428.692536404" Nov 03 20:21:27.970972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:27.970923 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" podStartSLOduration=1.970912728 podStartE2EDuration="1.970912728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:27.965782152 +0000 UTC m=+1428.690789749" watchObservedRunningTime="2026-11-03 20:21:27.970912728 +0000 UTC m=+1428.695920323" Nov 03 20:21:28.672049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:28.672021 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:28.743095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:28.743062 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:21:29.383462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:29.383417 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:30.080700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:30.080658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:30.273249 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:30.273213 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:21:30.805068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:30.805034 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:31.494914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:31.494869 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:32.177953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:32.177911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:32.844530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:32.844495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.108693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:33.108650 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.213866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:33.213771 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.517849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:33.517759 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.929827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:33.929787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.439812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:34.439770 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.813187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:34.807716 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.975528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:34.975417 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:21:35.016093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.016058 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:21:35.016312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.016224 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.021366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.021339 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6cf8ae-openshift-default-dockercfg-db88k\"" Nov 03 20:21:35.027345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.025622 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:35.097654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.097561 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.097654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.097617 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rs8\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.097855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.097757 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.097911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.097878 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.097978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.097941 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.098140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.098125 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.098221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.098153 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.098276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.098258 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.098322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.098308 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.133079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.133016 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:35.203857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.203816 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.211915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.211882 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.212545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.212517 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.213679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.213658 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.214302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.214284 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.214799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.214777 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rs8\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.215410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.215350 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.216178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.216156 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.216922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.216904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.217511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.215906 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.217661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.214157 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.217760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.216811 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.217866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.213491 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.217963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.217286 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.282524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.282487 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.285581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.285549 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.291694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.291663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.292976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.292906 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rs8\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.336190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.336155 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:35.828927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.828901 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:21:35.830710 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:35.830683 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f6a4cc_ae68_474e_81ba_7e1675060159.slice/crio-f2ccb153196eaebd8b412663369cc3072dcf3dd85e33735f6f48cc10c402b445 WatchSource:0}: Error finding container f2ccb153196eaebd8b412663369cc3072dcf3dd85e33735f6f48cc10c402b445: Status 404 returned error can't find the container with id f2ccb153196eaebd8b412663369cc3072dcf3dd85e33735f6f48cc10c402b445 Nov 03 20:21:35.833599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.833544 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:35.833664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.833635 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:35.833714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.833666 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:35.891824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:35.891800 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:36.058639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.058606 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerStarted","Data":"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217"} Nov 03 20:21:36.059036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.058666 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerStarted","Data":"f2ccb153196eaebd8b412663369cc3072dcf3dd85e33735f6f48cc10c402b445"} Nov 03 20:21:36.086029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.085925 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podStartSLOduration=2.085913025 podStartE2EDuration="2.085913025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:36.081195938 +0000 UTC m=+1436.806203535" watchObservedRunningTime="2026-11-03 20:21:36.085913025 +0000 UTC m=+1436.810920621" Nov 03 20:21:36.344177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.338364 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:36.344177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.338485 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:21:36.344177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:36.338529 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:21:37.028965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:37.028937 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:37.342554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:37.342471 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:37.634568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:37.634537 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:37.738558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:37.738522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.075348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.075275 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:38.076341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.076321 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:21:38.144498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.144462 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.343271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.343177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.444147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.444069 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.956161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:38.956126 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:21:39.108486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:39.108456 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.551097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:39.551055 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.666532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:39.666494 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.879486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:39.879450 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.981375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:39.981342 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:21:40.271899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.271815 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:21:40.276880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.276857 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.287752 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:21:40.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.287977 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e50c04-serving-cert\"" Nov 03 20:21:40.353932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.353899 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.372219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.372190 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.372381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.372325 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.372438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.372377 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjq7\" (UniqueName: \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.372563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.372551 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.471133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.470817 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.481635 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.481825 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.481968 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.482054 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjq7\" (UniqueName: \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:40.482968 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-serving-cert: secret "maas-api-e2e-api-b-e50c04-serving-cert" not found Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:21:40.483033 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls podName:e6fd72e2-20fd-4682-a18a-b96f3b327e41 nodeName:}" failed. No retries permitted until 2026-11-03 20:21:40.983011466 +0000 UTC m=+1441.708019058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls") pod "maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" (UID: "e6fd72e2-20fd-4682-a18a-b96f3b327e41") : secret "maas-api-e2e-api-b-e50c04-serving-cert" not found Nov 03 20:21:40.486125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.484099 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.517040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.516969 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.522861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.519335 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjq7\" (UniqueName: \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:40.692848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.692795 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.757969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.757936 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:21:40.763154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.763133 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.777458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.777431 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:21:40.796452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.796415 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.815207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.815174 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-683ba6-openshift-default-dockercfg-qlsk5\"" Nov 03 20:21:40.889926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.889890 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890055 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890256 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcgn\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890280 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890371 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890460 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890509 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890541 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.890704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.890688 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.895463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.895442 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.993466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.993425 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.994654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.994629 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.995012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.994988 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcgn\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.995189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.995171 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.995345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.995324 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:40.995479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.995462 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.007131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.995599 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.014218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.014181 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.017161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.017130 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.017427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.017408 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:41.017719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.008969 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.017848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.007757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.017952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.007998 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.018271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:40.994461 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.019053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.019031 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.025252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.019957 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.025252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.024769 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.035896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.035866 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") pod \"maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:41.040169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.036545 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.040169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.037814 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcgn\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\") pod \"e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.083676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.083408 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:41.198454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.198421 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:41.603798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.603765 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.685157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.685127 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:21:41.685753 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:41.685727 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e8413f_16d8_42b3_a5d1_517199912fc0.slice/crio-3f064bd307514f5788117d40046cbee6981a50b0cb6406f710f461eac7e373df WatchSource:0}: Error finding container 3f064bd307514f5788117d40046cbee6981a50b0cb6406f710f461eac7e373df: Status 404 returned error can't find the container with id 3f064bd307514f5788117d40046cbee6981a50b0cb6406f710f461eac7e373df Nov 03 20:21:41.689077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.689040 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:41.689172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.689144 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:41.689212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.689191 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:41.717359 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:21:41.708273 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fd72e2_20fd_4682_a18a_b96f3b327e41.slice/crio-19216e352ebb27616ec2f2fc705d2dbe9e87a8c6bbfe1a187f0f8e9b81e2d99d WatchSource:0}: Error finding container 19216e352ebb27616ec2f2fc705d2dbe9e87a8c6bbfe1a187f0f8e9b81e2d99d: Status 404 returned error can't find the container with id 19216e352ebb27616ec2f2fc705d2dbe9e87a8c6bbfe1a187f0f8e9b81e2d99d Nov 03 20:21:41.717359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:41.709052 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:21:42.181948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:42.181863 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerStarted","Data":"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1"} Nov 03 20:21:42.182153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:42.181974 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerStarted","Data":"3f064bd307514f5788117d40046cbee6981a50b0cb6406f710f461eac7e373df"} Nov 03 20:21:42.183529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:42.183501 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" event={"ID":"e6fd72e2-20fd-4682-a18a-b96f3b327e41","Type":"ContainerStarted","Data":"19216e352ebb27616ec2f2fc705d2dbe9e87a8c6bbfe1a187f0f8e9b81e2d99d"} Nov 03 20:21:42.207057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:42.207009 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podStartSLOduration=2.20699578 podStartE2EDuration="2.20699578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:42.205820703 +0000 UTC m=+1442.930828296" watchObservedRunningTime="2026-11-03 20:21:42.20699578 +0000 UTC m=+1442.932003376" Nov 03 20:21:42.598847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:42.598809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:43.084175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.084131 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:43.090206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.090181 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:43.275144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.271585 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" event={"ID":"e6fd72e2-20fd-4682-a18a-b96f3b327e41","Type":"ContainerStarted","Data":"dfee5a99158f8f740e2af2a0be4334649d15a47ed14c32319911a144f74e4813"} Nov 03 20:21:43.275144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.271665 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:43.275144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.272009 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:43.362186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.348489 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:21:43.362186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.354940 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" podStartSLOduration=2.7991043859999998 podStartE2EDuration="3.354920301s" totalImagesPullingTime="555.815915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:43.353461074 +0000 UTC m=+1444.078468707" watchObservedRunningTime="2026-11-03 20:21:43.354920301 +0000 UTC m=+1444.079927899" Nov 03 20:21:43.787657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:43.787580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.625571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.625536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.731438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.731399 5954 scope.go:122] "RemoveContainer" containerID="6541b08fe31c204e965a285cfe5188c17cdf494e80eee71cebed3fbf138bde66" Nov 03 20:21:44.733772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.733739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.744509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.744485 5954 scope.go:122] "RemoveContainer" containerID="5361cf7d72fed7bb83566bbd8ef344c2c2a07231089756c9863fad3a01e7d02d" Nov 03 20:21:44.755473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.755431 5954 scope.go:122] "RemoveContainer" containerID="1200604f83c13c671cfc1cf0c304a3866e44a54198d025b389882c3fbd523a87" Nov 03 20:21:44.769822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.769798 5954 scope.go:122] "RemoveContainer" containerID="1335d8d1f7d04a5034124c07249f4126a9db4c3905fc478ea2404eccb33086e2" Nov 03 20:21:44.782448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.782422 5954 scope.go:122] "RemoveContainer" containerID="87dca1abe99215a99557a310eb794aa820b98a02487cd1582af6b9aaadb5812a" Nov 03 20:21:44.796401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.796381 5954 scope.go:122] "RemoveContainer" containerID="5d2fc9bf413dbdcb0dce4b849c874026f0416adfcc60ef7885808d7cdd41d62b" Nov 03 20:21:44.812297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:44.810955 5954 scope.go:122] "RemoveContainer" containerID="d40f45d5aa5bc543a0d706d3baf0690a5aeb2a6f0cf7c38744be68ff5433d774" Nov 03 20:21:45.014140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:45.011821 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:45.127810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:45.127775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:45.709476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:45.709388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:46.202376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:46.201942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:46.340144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:46.337752 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:46.912673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:46.912602 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:47.701119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:47.701078 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:48.037965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:48.037845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:48.477844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:48.477777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.140508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:49.140473 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.888976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:49.888944 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:50.399973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:50.399947 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:21:50.689273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:50.689189 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:51.192256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.192214 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:21:51.192768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.192727 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" podUID="7e6feb82-f483-403f-885c-09881e53bae5" containerName="maas-api" containerID="cri-o://716546d420d1ab1a990348d27a204cd9ef3868137e3845609d350c4bad33c23c" gracePeriod=30 Nov 03 20:21:51.246051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.244579 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:51.428135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.424100 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:51.545087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.545048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:51.963414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.963264 5954 generic.go:356] "Generic (PLEG): container finished" podID="7e6feb82-f483-403f-885c-09881e53bae5" containerID="716546d420d1ab1a990348d27a204cd9ef3868137e3845609d350c4bad33c23c" exitCode=0 Nov 03 20:21:51.963707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.963680 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" event={"ID":"7e6feb82-f483-403f-885c-09881e53bae5","Type":"ContainerDied","Data":"716546d420d1ab1a990348d27a204cd9ef3868137e3845609d350c4bad33c23c"} Nov 03 20:21:51.971198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:51.971163 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:52.070693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.070345 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:21:52.169508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.169466 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\") pod \"7e6feb82-f483-403f-885c-09881e53bae5\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " Nov 03 20:21:52.169701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.169551 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\" (UniqueName: \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\") pod \"7e6feb82-f483-403f-885c-09881e53bae5\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " Nov 03 20:21:52.169701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.169657 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\") pod \"7e6feb82-f483-403f-885c-09881e53bae5\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " Nov 03 20:21:52.169701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.169675 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\") pod \"7e6feb82-f483-403f-885c-09881e53bae5\" (UID: \"7e6feb82-f483-403f-885c-09881e53bae5\") " Nov 03 20:21:52.169922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.169885 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca" pod "7e6feb82-f483-403f-885c-09881e53bae5" (UID: "7e6feb82-f483-403f-885c-09881e53bae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:52.170471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.170449 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6feb82-f483-403f-885c-09881e53bae5-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:52.172340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.172302 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs" pod "7e6feb82-f483-403f-885c-09881e53bae5" (UID: "7e6feb82-f483-403f-885c-09881e53bae5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:52.172460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.172350 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls" pod "7e6feb82-f483-403f-885c-09881e53bae5" (UID: "7e6feb82-f483-403f-885c-09881e53bae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:52.172460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.172352 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk" pod "7e6feb82-f483-403f-885c-09881e53bae5" (UID: "7e6feb82-f483-403f-885c-09881e53bae5"). InnerVolumeSpecName "kube-api-access-mhzgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:52.271727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.271651 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:52.271727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.271676 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e6feb82-f483-403f-885c-09881e53bae5-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:52.271727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.271686 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzgk\" (UniqueName: \"kubernetes.io/projected/7e6feb82-f483-403f-885c-09881e53bae5-kube-api-access-mhzgk\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:52.784155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.784117 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:52.976501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.976467 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" Nov 03 20:21:52.976501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.976483 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m" event={"ID":"7e6feb82-f483-403f-885c-09881e53bae5","Type":"ContainerDied","Data":"823197f7506d36cea0048d72bf1f3ef924cad47bd92d08d3496c1a40cef37b29"} Nov 03 20:21:52.976765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:52.976549 5954 scope.go:122] "RemoveContainer" containerID="716546d420d1ab1a990348d27a204cd9ef3868137e3845609d350c4bad33c23c" Nov 03 20:21:53.008582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:53.008516 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:21:53.014116 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:53.014076 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-n894m"] Nov 03 20:21:53.409643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:53.409605 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6feb82-f483-403f-885c-09881e53bae5" path="/var/lib/kubelet/pods/7e6feb82-f483-403f-885c-09881e53bae5/volumes" Nov 03 20:21:53.559878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:53.559838 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.937875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:53.937837 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.037714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:54.037672 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.534884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:54.534839 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.221960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.221921 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:21:55.222936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.222913 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7e6feb82-f483-403f-885c-09881e53bae5" containerName="maas-api" Nov 03 20:21:55.223049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.222938 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6feb82-f483-403f-885c-09881e53bae5" containerName="maas-api" Nov 03 20:21:55.223180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.223163 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6feb82-f483-403f-885c-09881e53bae5" containerName="maas-api" Nov 03 20:21:55.275546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.275495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.284488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.284437 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:21:55.284715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.284546 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.289154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.288905 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6cf8ae-serving-cert\"" Nov 03 20:21:55.446134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.430090 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg46k\" (UniqueName: \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.446134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.430179 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.446134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.430258 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.446134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.430395 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.452732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.450526 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.533212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.533093 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.533388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.533217 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.533459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.533409 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.533736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.533714 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg46k\" (UniqueName: \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.533882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.533861 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.536509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.536483 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.537748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.537724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.546508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.546479 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg46k\" (UniqueName: \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\") pod \"maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.623765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.623727 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:55.758148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:55.753639 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.125494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.115801 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:21:56.212181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.210858 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.757544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.757472 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:21:56.758326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.757733 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" containerID="cri-o://17416d6823128370bb5df90ac6a170068364af759281ce2513f5cd32844f3834" gracePeriod=30 Nov 03 20:21:56.761652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.761622 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:21:56.761838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:56.761816 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" containerID="cri-o://49c6aaa68a56e1565c5268714831a5498270095dde1fa940770a9b5f910dad2c" gracePeriod=30 Nov 03 20:21:57.066916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.066867 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerStarted","Data":"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298"} Nov 03 20:21:57.067089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.066955 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:21:57.067089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.066982 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerStarted","Data":"0fade55172d8a73a385011698e4c8f2f2be12933a11d7db9008aa096255cd9c1"} Nov 03 20:21:57.092727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.092675 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" podStartSLOduration=1.625169953 podStartE2EDuration="2.092654459s" totalImagesPullingTime="467.484506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:57.088709845 +0000 UTC m=+1457.813717443" watchObservedRunningTime="2026-11-03 20:21:57.092654459 +0000 UTC m=+1457.817662054" Nov 03 20:21:57.133330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.133286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.812083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.812006 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:57.812513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:57.812076 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:21:58.025926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.025894 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:58.134862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.134364 5954 generic.go:356] "Generic (PLEG): container finished" podID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerID="4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16" exitCode=0 Nov 03 20:21:58.134862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.134492 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerDied","Data":"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16"} Nov 03 20:21:58.135212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.135053 5954 scope.go:122] "RemoveContainer" containerID="4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16" Nov 03 20:21:58.153057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.146298 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:58.153057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.146403 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:58.153057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.146447 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:58.203278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.203178 5954 generic.go:356] "Generic (PLEG): container finished" podID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerID="f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9" exitCode=0 Nov 03 20:21:58.215127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.204452 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerDied","Data":"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9"} Nov 03 20:21:58.215127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.205276 5954 scope.go:122] "RemoveContainer" containerID="f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9" Nov 03 20:21:58.215127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.207456 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:58.215127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.208209 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:58.215127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:58.208268 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:59.191275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.191243 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:59.214062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.214030 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerStarted","Data":"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8"} Nov 03 20:21:59.216331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.216305 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerStarted","Data":"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896"} Nov 03 20:21:59.316556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.316497 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:59.380185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.380159 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:21:59.380617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.380535 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:59.380778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.380754 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:21:59.874256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.874218 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:59.991623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:21:59.991589 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.119130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.114671 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:22:00.131749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.131667 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.132497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.132467 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:00.387030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.386511 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.387030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.386575 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:00.500761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.500688 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.569501 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:00.569456 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:22:00.616133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:00.605849 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.099845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.098511 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.099845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.098582 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:01.379327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.379296 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:22:01.384240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.383825 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.384240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.383879 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:01.385089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.385066 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.390278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.390253 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:22:01.390666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.390373 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.402157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.402128 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-683ba6-serving-cert\"" Nov 03 20:22:01.483887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.483852 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhdm\" (UniqueName: \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.484066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.484040 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.484196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.484173 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.484283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.484270 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.484917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.484865 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.640193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.630841 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhdm\" (UniqueName: \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.640193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.630915 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.640193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.631018 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.640193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.631164 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.640193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.636096 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.708171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.687776 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.782243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.779660 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhdm\" (UniqueName: \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:01.813230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:01.810902 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:02.073444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.073406 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:02.134464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.115195 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.134464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.115250 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:02.595204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.590688 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.595204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.590735 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:02.595204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.593475 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:02.704179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.698813 5954 generic.go:356] "Generic (PLEG): container finished" podID="4458b154-86c1-437c-9e9c-539f67f37f85" containerID="17416d6823128370bb5df90ac6a170068364af759281ce2513f5cd32844f3834" exitCode=0 Nov 03 20:22:02.708183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.705285 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" event={"ID":"4458b154-86c1-437c-9e9c-539f67f37f85","Type":"ContainerDied","Data":"17416d6823128370bb5df90ac6a170068364af759281ce2513f5cd32844f3834"} Nov 03 20:22:02.755158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.752263 5954 generic.go:356] "Generic (PLEG): container finished" podID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerID="49c6aaa68a56e1565c5268714831a5498270095dde1fa940770a9b5f910dad2c" exitCode=0 Nov 03 20:22:02.755158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:02.752473 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" event={"ID":"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac","Type":"ContainerDied","Data":"49c6aaa68a56e1565c5268714831a5498270095dde1fa940770a9b5f910dad2c"} Nov 03 20:22:03.107444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.097068 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:03.107444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.097145 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:03.113929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.113892 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:03.153130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.150297 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:22:03.262319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.262227 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:22:03.302287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.302248 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": dial tcp 10.134.0.146:8443: connect: connection refused" Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329163 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329228 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329325 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329431 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329482 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329555 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329662 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.332887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.332857 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.349709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.329700 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.349709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.341332 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\") pod \"4458b154-86c1-437c-9e9c-539f67f37f85\" (UID: \"4458b154-86c1-437c-9e9c-539f67f37f85\") " Nov 03 20:22:03.406644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.406125 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.410413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.408417 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:03.484205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.470338 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.489382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489345 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.489706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489683 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.489786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489714 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.489786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489732 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4458b154-86c1-437c-9e9c-539f67f37f85-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.489786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489756 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.489786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489773 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.490024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489852 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:03.490024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.489900 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:03.495396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.495366 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.500289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.500259 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:03.512653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.512580 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:03.513558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.513515 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.585440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.585369 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz" pod "4458b154-86c1-437c-9e9c-539f67f37f85" (UID: "4458b154-86c1-437c-9e9c-539f67f37f85"). InnerVolumeSpecName "kube-api-access-49jqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.586268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.586242 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:22:03.637180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.619315 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4458b154-86c1-437c-9e9c-539f67f37f85-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.665184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.665010 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jqz\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-kube-api-access-49jqz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.665184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.665046 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4458b154-86c1-437c-9e9c-539f67f37f85-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.665184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.665073 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4458b154-86c1-437c-9e9c-539f67f37f85-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.767345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.767262 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.786372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.786341 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.786642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.786623 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.786811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.786793 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.786986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.786968 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.787253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.787235 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.787431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.787415 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.787591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.787576 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.787886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.787866 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\") pod \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\" (UID: \"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac\") " Nov 03 20:22:03.797365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.770607 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.797365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.794542 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.797365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.794899 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.797365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.796295 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:03.865344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.865303 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:03.865706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.865684 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "kube-api-access-w4hr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.877619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.875348 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.877785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.875423 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.887168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.883314 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy" pod "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" (UID: "1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:03.887168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.887102 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:22:03.893636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.893457 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" Nov 03 20:22:03.893636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.893477 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc" event={"ID":"4458b154-86c1-437c-9e9c-539f67f37f85","Type":"ContainerDied","Data":"ff1e08d44818324451e4487f3ad1386f22060be335c0f08b941f7d9b81cee3e3"} Nov 03 20:22:03.893636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.893535 5954 scope.go:122] "RemoveContainer" containerID="17416d6823128370bb5df90ac6a170068364af759281ce2513f5cd32844f3834" Nov 03 20:22:03.896620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.896577 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" event={"ID":"de8d6ef3-890b-4146-95a1-f9367f7194b0","Type":"ContainerStarted","Data":"5e36f43f5d940d35e2bbdd5d9b4c921de84c53ad5ce39bd69cba04a62cefd2af"} Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897557 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897576 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897587 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897605 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897615 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897623 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hr4\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-kube-api-access-w4hr4\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897634 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897642 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.897709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.897652 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.898762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.898747 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" Nov 03 20:22:03.898934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.898752 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4" event={"ID":"1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac","Type":"ContainerDied","Data":"9ed9f722f745942a004a4d8016d0170b0ddfa25921abca610caebf7b4c5b6783"} Nov 03 20:22:03.910009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.909210 5954 scope.go:122] "RemoveContainer" containerID="49c6aaa68a56e1565c5268714831a5498270095dde1fa940770a9b5f910dad2c" Nov 03 20:22:03.983152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.979422 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:22:03.983152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.981754 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-bbjcc"] Nov 03 20:22:03.998593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:03.998399 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:22:04.002841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.002813 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9ckk4"] Nov 03 20:22:04.098409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.098329 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:04.098409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.098382 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:04.281572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.281542 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.382127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.381655 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:04.382127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.381746 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:04.811722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.811692 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:22:04.812216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.811878 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" podUID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" containerName="maas-api" containerID="cri-o://6199ed590476b57240b585cab845b8f24197f877ffb8986344eda332e64462f8" gracePeriod=30 Nov 03 20:22:04.877130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.876206 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.978184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.968944 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" event={"ID":"de8d6ef3-890b-4146-95a1-f9367f7194b0","Type":"ContainerStarted","Data":"c14f9ae5b2d0789899f9bf539417285c2774d0855dcb402868d029f2a24b4549"} Nov 03 20:22:04.978184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:04.972207 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:05.009138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.009045 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" podStartSLOduration=3.506060634 podStartE2EDuration="4.009023378s" totalImagesPullingTime="502.962744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:04.996468429 +0000 UTC m=+1465.721476019" watchObservedRunningTime="2026-11-03 20:22:05.009023378 +0000 UTC m=+1465.734030975" Nov 03 20:22:05.036877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.036784 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.100657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.100620 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:05.102382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.102338 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:05.140144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.138566 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.385682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.360829 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.385682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.381803 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:05.385682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.381864 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:05.518131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.517485 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" path="/var/lib/kubelet/pods/1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac/volumes" Nov 03 20:22:05.522130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.518353 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" path="/var/lib/kubelet/pods/4458b154-86c1-437c-9e9c-539f67f37f85/volumes" Nov 03 20:22:05.622130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.596510 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.984564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.984529 5954 generic.go:356] "Generic (PLEG): container finished" podID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" containerID="6199ed590476b57240b585cab845b8f24197f877ffb8986344eda332e64462f8" exitCode=0 Nov 03 20:22:05.985040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:05.984596 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" event={"ID":"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb","Type":"ContainerDied","Data":"6199ed590476b57240b585cab845b8f24197f877ffb8986344eda332e64462f8"} Nov 03 20:22:06.094321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.094296 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:22:06.097398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.097374 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 20:22:06.097476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.097417 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 20:22:06.281701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.281600 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\") pod \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " Nov 03 20:22:06.283691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.283485 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\" (UniqueName: \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\") pod \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " Nov 03 20:22:06.283869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.283763 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\") pod \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " Nov 03 20:22:06.283869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.283804 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\") pod \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\" (UID: \"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb\") " Nov 03 20:22:06.289004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.288265 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs" pod "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" (UID: "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:06.289004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.288937 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca" pod "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" (UID: "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:06.289294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.289255 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls" pod "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" (UID: "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:06.289426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.289403 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4" pod "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" (UID: "bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb"). InnerVolumeSpecName "kube-api-access-gb2r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:06.379939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.379906 5954 patch_prober.go:29] interesting pod/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:22:06.380119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.379968 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 20:22:06.386339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.386315 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb2r4\" (UniqueName: \"kubernetes.io/projected/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-kube-api-access-gb2r4\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:06.386339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.386339 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:06.386448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.386349 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:06.386448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.386359 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:06.554923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.554843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:06.995074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.994723 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" Nov 03 20:22:06.995074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.994745 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4" event={"ID":"bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb","Type":"ContainerDied","Data":"a15dce9459824e8a6e94adc250e301142f5b83a7dbe8927e30a310a265bed60c"} Nov 03 20:22:06.995074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.994822 5954 scope.go:122] "RemoveContainer" containerID="6199ed590476b57240b585cab845b8f24197f877ffb8986344eda332e64462f8" Nov 03 20:22:06.996710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.996683 5954 generic.go:356] "Generic (PLEG): container finished" podID="16787376-4114-4876-8e0a-cd916a455f71" containerID="ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298" exitCode=1 Nov 03 20:22:06.996861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.996715 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerDied","Data":"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298"} Nov 03 20:22:06.997147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:06.997130 5954 scope.go:122] "RemoveContainer" containerID="ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298" Nov 03 20:22:07.041578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.041540 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:22:07.045299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.045272 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06294c6a-5f9dd8497-l6lx4"] Nov 03 20:22:07.104053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.104017 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:22:07.106358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.106331 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:22:07.360644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.360609 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.384978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.384948 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:22:07.385456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.385398 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:22:07.386641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.386618 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:22:07.552123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:07.552079 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" path="/var/lib/kubelet/pods/bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb/volumes" Nov 03 20:22:08.014229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:08.013586 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerStarted","Data":"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3"} Nov 03 20:22:08.015139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:08.014902 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:22:08.045175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:08.045139 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:08.249129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:08.237967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:08.847395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:08.847364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.564783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.564726 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:22:09.565691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565667 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" Nov 03 20:22:09.565691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565691 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" Nov 03 20:22:09.565872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565723 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" containerName="maas-api" Nov 03 20:22:09.565872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565732 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" containerName="maas-api" Nov 03 20:22:09.565872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565784 5954 cpu_manager.go:414] "containerMap: removing container" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" Nov 03 20:22:09.565872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565792 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" Nov 03 20:22:09.566071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.565990 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3fc6ac-d233-430f-9bd7-5af3bb4e1bcb" containerName="maas-api" Nov 03 20:22:09.566071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.566015 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac8ca03-d913-4b2e-97f8-c1bd5fa581ac" containerName="istio-proxy" Nov 03 20:22:09.566071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.566037 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4458b154-86c1-437c-9e9c-539f67f37f85" containerName="istio-proxy" Nov 03 20:22:09.569985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.569960 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.586693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.586665 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-15e02e6b-serving-cert\"" Nov 03 20:22:09.593430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.593398 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:22:09.629769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.629735 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.653562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.653533 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.653778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.653760 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.653830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.653804 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b792z\" (UniqueName: \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.654038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.654019 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.737132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.732586 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.758252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.758212 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.758444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.758283 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.758556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.758533 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.758619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.758598 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b792z\" (UniqueName: \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.765606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.759762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.765606 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:09.760292 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-serving-cert: secret "maas-api-e2e-mt-15e02e6b-serving-cert" not found Nov 03 20:22:09.765606 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:09.760354 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls podName:d407f0c7-cf54-41c2-ab04-f2d71dff3394 nodeName:}" failed. No retries permitted until 2026-11-03 20:22:10.2603349 +0000 UTC m=+1470.985342476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls") pod "maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" (UID: "d407f0c7-cf54-41c2-ab04-f2d71dff3394") : secret "maas-api-e2e-mt-15e02e6b-serving-cert" not found Nov 03 20:22:09.765606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.764968 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.777539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.774514 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b792z\" (UniqueName: \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:09.844677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:09.844586 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.278130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:10.275070 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:10.307131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:10.306566 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") pod \"maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:10.494396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:10.494359 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:10.660385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:10.660342 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.935149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:10.934589 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:22:11.003037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:11.002369 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:22:11.057267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:11.057220 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" event={"ID":"d407f0c7-cf54-41c2-ab04-f2d71dff3394","Type":"ContainerStarted","Data":"b38a02ddad69f60b33599add9f93f335ed0a58c4a68f9a31b257d33c267c6ac9"} Nov 03 20:22:11.546391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:11.546349 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:11.894782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:11.894714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:11.995906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:11.995857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:12.067778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:12.067740 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" event={"ID":"d407f0c7-cf54-41c2-ab04-f2d71dff3394","Type":"ContainerStarted","Data":"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d"} Nov 03 20:22:12.067956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:12.067871 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:12.116956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:12.116890 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" podStartSLOduration=2.7355212140000003 podStartE2EDuration="3.116867305s" totalImagesPullingTime="381.346091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:12.094604011 +0000 UTC m=+1472.819611608" watchObservedRunningTime="2026-11-03 20:22:12.116867305 +0000 UTC m=+1472.841874902" Nov 03 20:22:12.583616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:12.583580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.183267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:13.183235 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.287135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:13.286253 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.407245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:13.407214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.520767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:13.520672 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:22:13.559507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:13.557643 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.027946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.027917 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:22:14.099790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.099752 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" containerID="cri-o://c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8" gracePeriod=30 Nov 03 20:22:14.440558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.440520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.534778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.534746 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.753099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.753015 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:22:14.758989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.758969 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.764121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.764092 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3ebde5-openshift-default-dockercfg-mmkph\"" Nov 03 20:22:14.771175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.771156 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:22:14.793012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.792961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.834725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.834609 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.834893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.834745 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.834893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.834780 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.834979 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.835091 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.835157 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.835348 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.835387 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r2s\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.835464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.835451 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.901231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.901197 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.939730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.939690 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.940472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.940230 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r2s\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.940958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.940852 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.941582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.941534 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942102 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942165 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942199 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942306 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942364 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942399 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.943142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.942945 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.944083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.943742 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.945057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.945037 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.946324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.946304 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.967253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.967169 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:14.968868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.968842 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:15.013180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:14.995229 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r2s\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:15.013180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.004748 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\") pod \"e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:15.105188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.104603 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:15.832062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.831977 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:15.863932 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:22:15.863901 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493a7ba1_730d_4156_aee0_91c7865fd71a.slice/crio-d7bdd933b20404f03c1a7a4b2e9e770fbb2bec8c5e402d95b7f0c1d7b18af24e WatchSource:0}: Error finding container d7bdd933b20404f03c1a7a4b2e9e770fbb2bec8c5e402d95b7f0c1d7b18af24e: Status 404 returned error can't find the container with id d7bdd933b20404f03c1a7a4b2e9e770fbb2bec8c5e402d95b7f0c1d7b18af24e Nov 03 20:22:15.864084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.864046 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:22:15.866941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.866908 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:15.867031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.866969 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:15.867031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:15.867008 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:16.142277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:16.142241 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerStarted","Data":"b70efb7085501d9437bfe3dd1ef78114c4a58bc9ccf3affdd3365eca107ae7d4"} Nov 03 20:22:16.142487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:16.142299 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerStarted","Data":"d7bdd933b20404f03c1a7a4b2e9e770fbb2bec8c5e402d95b7f0c1d7b18af24e"} Nov 03 20:22:16.171405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:16.171355 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podStartSLOduration=2.171341624 podStartE2EDuration="2.171341624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:16.167847218 +0000 UTC m=+1476.892854815" watchObservedRunningTime="2026-11-03 20:22:16.171341624 +0000 UTC m=+1476.896349219" Nov 03 20:22:16.579181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:16.579062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:17.106170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:17.106132 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:17.111788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:17.111766 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:17.152843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:17.152812 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:17.153960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:17.153938 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:22:17.336773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:17.336743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:18.095821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:18.095792 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:22:18.224705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:18.224671 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:18.513297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:18.513257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:18.610561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:18.610525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.039245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.039214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.276323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.276286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.466646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.466500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.512234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.512208 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:22:19.539064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539032 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539168 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539209 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539260 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539333 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539372 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539399 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539540 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:19.539688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539642 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.539749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.539705 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\") pod \"d6d0c216-9984-461c-bd9a-72cca88ade5b\" (UID: \"d6d0c216-9984-461c-bd9a-72cca88ade5b\") " Nov 03 20:22:19.540477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.540446 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:19.540951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.540922 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.541046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.540955 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d0c216-9984-461c-bd9a-72cca88ade5b-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.541046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.540542 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:19.542238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.542195 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:19.542343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.542259 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:19.543552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.542897 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:19.544122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.544071 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:19.548101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.548078 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:19.548217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.548173 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw" pod "d6d0c216-9984-461c-bd9a-72cca88ade5b" (UID: "d6d0c216-9984-461c-bd9a-72cca88ade5b"). InnerVolumeSpecName "kube-api-access-p2svw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:19.642855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642816 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.642855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642847 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.642855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642858 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.643087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642867 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.643087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642877 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2svw\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-kube-api-access-p2svw\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.643087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642890 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d0c216-9984-461c-bd9a-72cca88ade5b-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.643087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.642903 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d0c216-9984-461c-bd9a-72cca88ade5b-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:19.777986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.777904 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.883164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:19.883076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:20.237229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.231888 5954 generic.go:356] "Generic (PLEG): container finished" podID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerID="c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8" exitCode=0 Nov 03 20:22:20.237229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.232216 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerDied","Data":"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8"} Nov 03 20:22:20.237229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.232288 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" event={"ID":"d6d0c216-9984-461c-bd9a-72cca88ade5b","Type":"ContainerDied","Data":"9aabd60bfc5ade4b5501d89270305b7b3d37205a8501ebbf835dd119b5ac1227"} Nov 03 20:22:20.237229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.232321 5954 scope.go:122] "RemoveContainer" containerID="c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8" Nov 03 20:22:20.237229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.232493 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj" Nov 03 20:22:20.275717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.275692 5954 scope.go:122] "RemoveContainer" containerID="d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769" Nov 03 20:22:20.286208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.286179 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:22:20.289873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.289852 5954 scope.go:122] "RemoveContainer" containerID="c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8" Nov 03 20:22:20.290244 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:20.290219 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8\": container with ID starting with c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8 not found: ID does not exist" containerID="c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8" Nov 03 20:22:20.290316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.290253 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8"} err="failed to get container status \"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8\": rpc error: code = NotFound desc = could not find container \"c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8\": container with ID starting with c5a2cfc95fd86f17b3cc15a1a4f059d600b117bde93ad66e20c6b78367e87bd8 not found: ID does not exist" Nov 03 20:22:20.290316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.290272 5954 scope.go:122] "RemoveContainer" containerID="d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769" Nov 03 20:22:20.290316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.290275 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06294c6a-openshift-default-85555b884f-67qxj"] Nov 03 20:22:20.290581 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:20.290562 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769\": container with ID starting with d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769 not found: ID does not exist" containerID="d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769" Nov 03 20:22:20.290623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.290588 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769"} err="failed to get container status \"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769\": rpc error: code = NotFound desc = could not find container \"d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769\": container with ID starting with d20d0ba0732bb70e0434f829ace759017fb7ed8f4dbb6143b7c44187028dd769 not found: ID does not exist" Nov 03 20:22:20.556570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:20.556461 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:21.493201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:21.493003 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" path="/var/lib/kubelet/pods/d6d0c216-9984-461c-bd9a-72cca88ade5b/volumes" Nov 03 20:22:21.881495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:21.881465 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:22.772761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:22.772723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.542803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:23.542767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.311076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.311043 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.412822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.412779 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.718494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.718465 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:22:24.719242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.719227 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:24.719303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.719243 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:24.719407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.719398 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:24.719445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.719413 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:24.725082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.725060 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.730735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.730507 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-77e83b-openshift-default-dockercfg-khxxv\"" Nov 03 20:22:24.745381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.745358 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:22:24.747657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747612 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.747657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747654 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.747852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747762 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.747852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747821 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.747932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747867 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.747976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.747929 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh99g\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.748185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.748162 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.748303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.748281 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.748361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.748346 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.770281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.770210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.931768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.931729 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.942743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.942713 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.944630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.944605 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.946489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.945821 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.949480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.949455 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.950550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.950532 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.951033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.951014 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.951551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.951524 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.952259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.952241 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh99g\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.953044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.945048 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.953665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.943493 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.953797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.951375 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.953886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.942563 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:24.953961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:24.950374 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.039186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.036832 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.039186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.037189 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.039186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.037850 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.055128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.051615 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh99g\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.278080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.278049 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:25.355178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.348427 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:25.526826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.526793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:25.696156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.696123 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:22:25.696393 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:22:25.696370 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33533d2a_ed7c_48a1_9fbb_aaaf2264b3ee.slice/crio-499694a301cc8dfcf81f976d6a6eaafa87fc837b9b0f61370da9d73a0e9077c8 WatchSource:0}: Error finding container 499694a301cc8dfcf81f976d6a6eaafa87fc837b9b0f61370da9d73a0e9077c8: Status 404 returned error can't find the container with id 499694a301cc8dfcf81f976d6a6eaafa87fc837b9b0f61370da9d73a0e9077c8 Nov 03 20:22:25.699594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.699556 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:25.699703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.699632 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:25.699703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:25.699667 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:26.186931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.186892 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:22:26.283345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.283311 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerStarted","Data":"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615"} Nov 03 20:22:26.283499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.283376 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerStarted","Data":"499694a301cc8dfcf81f976d6a6eaafa87fc837b9b0f61370da9d73a0e9077c8"} Nov 03 20:22:26.333127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.332431 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podStartSLOduration=2.332409029 podStartE2EDuration="2.332409029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:26.327509023 +0000 UTC m=+1487.052516646" watchObservedRunningTime="2026-11-03 20:22:26.332409029 +0000 UTC m=+1487.057416628" Nov 03 20:22:26.358509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.350937 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:26.409126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.408557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.578203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:26.577743 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:27.292274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:27.292246 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:27.293102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:27.293079 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:22:27.462341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:27.462302 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.580334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:28.580284 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.803645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:28.803609 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.937933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:28.937863 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.093672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:29.093408 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.544261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:29.544206 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.789131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:29.786704 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.033999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.031696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.180192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.180161 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.243995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.242472 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:22:30.243995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.243702 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:30.243995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.243725 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d0c216-9984-461c-bd9a-72cca88ade5b" containerName="istio-proxy" Nov 03 20:22:30.307902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.307781 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.310313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.310291 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.362130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.361437 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:22:30.699130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.685042 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ca474-openshift-default-dockercfg-p4nhm\"" Nov 03 20:22:30.808474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.785936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.823522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.823475 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.823830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.823809 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.823992 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.824220 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.824470 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95974\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.824621 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.824793 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.824947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.824931 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:30.825083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:30.825067 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.039704 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040064 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040194 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040395 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95974\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040446 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040623 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040738 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.041368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.040842 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.099139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.045570 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.099139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.053888 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.099139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.059651 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.157132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.153345 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.190322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.190286 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.288140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.240920 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.291180 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:22:31.291143 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:22:31.291367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.271401 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.291529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.241323 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.301002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.300972 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95974\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.301684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.301637 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:31.366809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.366262 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:22:31.574186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.574040 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:31.595955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:31.594431 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:32.592647 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:22:32.581388 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9081ec50_cc11_4232_af9c_178cacc31f6b.slice/crio-ac12b16f2a87836e651c0619cb881c2151b132cc4e2ae0232af7f467950ac798 WatchSource:0}: Error finding container ac12b16f2a87836e651c0619cb881c2151b132cc4e2ae0232af7f467950ac798: Status 404 returned error can't find the container with id ac12b16f2a87836e651c0619cb881c2151b132cc4e2ae0232af7f467950ac798 Nov 03 20:22:32.593982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:32.593926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:32.607019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:32.606978 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:32.607254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:32.607225 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:32.607441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:32.607420 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:32.612170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:32.611829 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:22:33.046364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:33.046322 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerStarted","Data":"ac12b16f2a87836e651c0619cb881c2151b132cc4e2ae0232af7f467950ac798"} Nov 03 20:22:33.278469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:33.278380 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.023016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.022971 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.137815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.137774 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerStarted","Data":"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7"} Nov 03 20:22:34.185132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.170684 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podStartSLOduration=4.170662791 podStartE2EDuration="4.170662791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:34.165613145 +0000 UTC m=+1494.890620786" watchObservedRunningTime="2026-11-03 20:22:34.170662791 +0000 UTC m=+1494.895670388" Nov 03 20:22:34.344185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.340185 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:34.600202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.599979 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:34.626133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.623425 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:34.809401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:34.809364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.241213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:35.241088 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:35.270135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:35.267401 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:22:35.938396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:35.938364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.404303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.404273 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:22:36.411815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.411791 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.422310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.422277 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:22:36.435526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.435491 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3ebde5-serving-cert\"" Nov 03 20:22:36.454050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.454013 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.454276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.454090 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.454276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.454159 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.454793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.454760 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v2f\" (UniqueName: \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.488953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.488901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.556134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.556027 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.556134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.556099 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.556357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.556152 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.556411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.556356 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v2f\" (UniqueName: \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.556950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.556918 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.559049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.559025 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.559480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.559456 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.569288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.569259 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v2f\" (UniqueName: \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\") pod \"maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.620508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.620288 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.757130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.748398 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:36.757130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.749522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.865959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:36.865356 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:37.034542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.032618 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:22:37.053418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.053378 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:22:37.053599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.053520 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.067430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.067397 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5de824-openshift-default-dockercfg-g4q4k\"" Nov 03 20:22:37.076879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.076847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:37.109990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.109936 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:22:37.117258 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:22:37.117197 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246e7f67_dfed_4abe_a72c_02e715aa9f48.slice/crio-6b28528f7e5cad0490e3b4b2a918dbbc06e3dd3ae5ffe2cdea47eec8bb04a9a4 WatchSource:0}: Error finding container 6b28528f7e5cad0490e3b4b2a918dbbc06e3dd3ae5ffe2cdea47eec8bb04a9a4: Status 404 returned error can't find the container with id 6b28528f7e5cad0490e3b4b2a918dbbc06e3dd3ae5ffe2cdea47eec8bb04a9a4 Nov 03 20:22:37.178996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.178959 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179029 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179101 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179198 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ctcx\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179299 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179479 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179514 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179560 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.179750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.179730 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290143 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290217 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290284 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290324 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ctcx\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290368 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290521 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290547 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290588 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.307133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.290746 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.385191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.313852 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.385191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.314571 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.385191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.314822 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.385191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.315048 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.385191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.315294 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.525906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.513755 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.540140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.536965 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.600689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.599852 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.600689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.600605 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ctcx\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.608843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.608795 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" event={"ID":"246e7f67-dfed-4abe-a72c-02e715aa9f48","Type":"ContainerStarted","Data":"6b28528f7e5cad0490e3b4b2a918dbbc06e3dd3ae5ffe2cdea47eec8bb04a9a4"} Nov 03 20:22:37.677615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.677512 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:37.798074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:37.797511 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.049855 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:22:38.049807 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd43e8ac_3fd7_49df_b061_f55be5c48375.slice/crio-8297cee85f9b125978f16d17340f09aea14b93ccd6f037e64f02c47eac058f52 WatchSource:0}: Error finding container 8297cee85f9b125978f16d17340f09aea14b93ccd6f037e64f02c47eac058f52: Status 404 returned error can't find the container with id 8297cee85f9b125978f16d17340f09aea14b93ccd6f037e64f02c47eac058f52 Nov 03 20:22:38.050063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.050029 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:22:38.054384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.054096 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.054384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.054193 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.054384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.054238 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.121441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.121345 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.261442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.256529 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.391908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.391823 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.617978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.617936 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" event={"ID":"246e7f67-dfed-4abe-a72c-02e715aa9f48","Type":"ContainerStarted","Data":"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd"} Nov 03 20:22:38.621217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.619961 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:38.629155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.627283 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerStarted","Data":"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8"} Nov 03 20:22:38.629155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.627347 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerStarted","Data":"8297cee85f9b125978f16d17340f09aea14b93ccd6f037e64f02c47eac058f52"} Nov 03 20:22:38.699222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.688311 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:38.763005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.762904 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" podStartSLOduration=2.179143679 podStartE2EDuration="2.762883184s" totalImagesPullingTime="583.739505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:38.684382785 +0000 UTC m=+1499.409390382" watchObservedRunningTime="2026-11-03 20:22:38.762883184 +0000 UTC m=+1499.487890783" Nov 03 20:22:38.765921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:38.765871 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podStartSLOduration=2.7658567720000002 podStartE2EDuration="2.765856772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:38.708891454 +0000 UTC m=+1499.433899051" watchObservedRunningTime="2026-11-03 20:22:38.765856772 +0000 UTC m=+1499.490864369" Nov 03 20:22:39.180190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.136206 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:39.180190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.136274 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:39.686753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.686718 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:39.719176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.717137 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.792608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.792247 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:39.792608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:39.792310 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:40.502019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:40.501637 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:40.650140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:40.650075 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:40.650913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:40.650888 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:22:40.751994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:40.751958 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:41.100639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:41.100591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:41.217083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:41.217046 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:41.676765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:41.676722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.406976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:42.406926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.524765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:42.524723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.639200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:42.637509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.555299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:43.550681 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.151631 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.264278 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.264356 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.737577 5954 generic.go:356] "Generic (PLEG): container finished" podID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerID="f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217" exitCode=0 Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.737649 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerDied","Data":"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217"} Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.738080 5954 scope.go:122] "RemoveContainer" containerID="f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217" Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.739857 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.739923 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:44.825904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.739971 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:44.951876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:44.951847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.507608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.507575 5954 scope.go:122] "RemoveContainer" containerID="d487014b98f9c40417d31bd6281e0d90df6af8da66c3ed588887dce3bbb573c7" Nov 03 20:22:45.546039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.546011 5954 scope.go:122] "RemoveContainer" containerID="785b2fbc83ab27cc94a5f5a0f0bbf020d92d768939e138eb669db7873009c2fa" Nov 03 20:22:45.556807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.556782 5954 scope.go:122] "RemoveContainer" containerID="6bc5bad2b4a3c524fcfdccd14869bcff790429f4e79ff236dfdad3f6337f7cef" Nov 03 20:22:45.656364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.656334 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:22:45.684406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.684377 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.754518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.754483 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerStarted","Data":"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e"} Nov 03 20:22:45.756547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.756523 5954 generic.go:356] "Generic (PLEG): container finished" podID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerID="2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1" exitCode=0 Nov 03 20:22:45.756689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.756588 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerDied","Data":"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1"} Nov 03 20:22:45.757095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.757072 5954 scope.go:122] "RemoveContainer" containerID="2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1" Nov 03 20:22:45.758957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.758934 5954 generic.go:356] "Generic (PLEG): container finished" podID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerID="5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e" exitCode=0 Nov 03 20:22:45.759051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.758970 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerDied","Data":"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e"} Nov 03 20:22:45.759245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.759218 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:45.759315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.759300 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:45.759357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.759342 5954 scope.go:122] "RemoveContainer" containerID="5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e" Nov 03 20:22:45.759391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.759353 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:45.761166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.761131 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:45.761251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.761205 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:45.761251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:45.761242 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:46.341130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.340401 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:22:46.341130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.340578 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:46.341130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.340665 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:46.530176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.530140 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:46.778975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.778938 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerStarted","Data":"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e"} Nov 03 20:22:46.787912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.786860 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerStarted","Data":"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729"} Nov 03 20:22:46.986337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:46.986241 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.084612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.084524 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:22:47.084956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.084934 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:47.085137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.085084 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:47.242130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.241615 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.338486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.338301 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:47.338486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.338376 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:47.580127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.579221 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:22:47.580127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.579602 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:47.580127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:47.579650 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:48.083457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.083338 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:48.095146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.094685 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:48.095146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.094759 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:48.219152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.212020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:48.382696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.382661 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:48.383000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.382960 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:48.585130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.578725 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:48.585130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.578788 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:48.983200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:48.983141 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:49.114316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.114271 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:49.114628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.114579 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:49.192641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.192506 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:49.370998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.370960 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:49.385244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.385192 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:49.606345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.606308 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:49.607388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.606962 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:49.984814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:49.984780 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:50.086943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.086492 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:50.086943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.086572 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:50.249337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.249242 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:50.298332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.298290 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:50.359013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.358976 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:50.359319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.359286 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:50.360311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.360283 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:50.580191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.579448 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:50.580191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.579526 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:50.636133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.613792 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:22:50.643130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.640524 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" podUID="8036b665-c559-434d-bb41-2e040e3ce501" containerName="maas-api" containerID="cri-o://135de9e3842b1ea71b1fec568f91e43742691a1b7a03a71fa4f8d9144924ef3b" gracePeriod=30 Nov 03 20:22:50.658063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:50.654786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.060132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.045293 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.202625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.121706 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:51.203046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.203012 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:51.364129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.342458 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.375270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.368683 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:51.375270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.368741 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:51.506941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.506904 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.511795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.511763 5954 generic.go:356] "Generic (PLEG): container finished" podID="8036b665-c559-434d-bb41-2e040e3ce501" containerID="135de9e3842b1ea71b1fec568f91e43742691a1b7a03a71fa4f8d9144924ef3b" exitCode=0 Nov 03 20:22:51.513433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.513398 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" event={"ID":"8036b665-c559-434d-bb41-2e040e3ce501","Type":"ContainerDied","Data":"135de9e3842b1ea71b1fec568f91e43742691a1b7a03a71fa4f8d9144924ef3b"} Nov 03 20:22:51.576698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.576600 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:51.576988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.576941 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:51.813477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:51.813440 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:52.112878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.111673 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:52.112878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.111738 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:52.402262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.402222 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 20:22:52.402470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.402292 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 20:22:52.402968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.402942 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.581533 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.581597 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.613229 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\") pod \"8036b665-c559-434d-bb41-2e040e3ce501\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.613583 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\") pod \"8036b665-c559-434d-bb41-2e040e3ce501\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.613627 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\") pod \"8036b665-c559-434d-bb41-2e040e3ce501\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " Nov 03 20:22:52.645128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.613678 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\" (UniqueName: \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\") pod \"8036b665-c559-434d-bb41-2e040e3ce501\" (UID: \"8036b665-c559-434d-bb41-2e040e3ce501\") " Nov 03 20:22:52.799130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.791425 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca" pod "8036b665-c559-434d-bb41-2e040e3ce501" (UID: "8036b665-c559-434d-bb41-2e040e3ce501"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:52.833130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.829544 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8036b665-c559-434d-bb41-2e040e3ce501-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.869361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.869324 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:52.870234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.870207 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls" pod "8036b665-c559-434d-bb41-2e040e3ce501" (UID: "8036b665-c559-434d-bb41-2e040e3ce501"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:52.870545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.870522 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5" pod "8036b665-c559-434d-bb41-2e040e3ce501" (UID: "8036b665-c559-434d-bb41-2e040e3ce501"). InnerVolumeSpecName "kube-api-access-2sqk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:52.893131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.890030 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs" pod "8036b665-c559-434d-bb41-2e040e3ce501" (UID: "8036b665-c559-434d-bb41-2e040e3ce501"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:52.902730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.902701 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" Nov 03 20:22:52.903220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.902730 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78" event={"ID":"8036b665-c559-434d-bb41-2e040e3ce501","Type":"ContainerDied","Data":"707e8399e42a7884d64f97d0518d3c0d8531a7027d4328eeb5c37308feadab8c"} Nov 03 20:22:52.903384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.903269 5954 scope.go:122] "RemoveContainer" containerID="135de9e3842b1ea71b1fec568f91e43742691a1b7a03a71fa4f8d9144924ef3b" Nov 03 20:22:52.958147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.957839 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.958147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.957882 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8036b665-c559-434d-bb41-2e040e3ce501-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.958147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.957899 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqk5\" (UniqueName: \"kubernetes.io/projected/8036b665-c559-434d-bb41-2e040e3ce501-kube-api-access-2sqk5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:52.990852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.990811 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:22:52.998227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:52.998186 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d5927-5f79fb78bb-qvb78"] Nov 03 20:22:53.083221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.082492 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:22:53.087493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.087462 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:53.087662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.087518 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:53.356134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.352924 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:22:53.407777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.407745 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8036b665-c559-434d-bb41-2e040e3ce501" path="/var/lib/kubelet/pods/8036b665-c559-434d-bb41-2e040e3ce501/volumes" Nov 03 20:22:53.575495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.575461 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:22:53.575765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.575745 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 03 20:22:53.725394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.724807 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:22:53.792557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:53.792521 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:54.010060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.009981 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:22:54.086486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.086455 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:22:54.086733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.086704 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 20:22:54.580122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.580085 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:22:54.581048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.581029 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:22:54.801982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:54.801938 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:55.057219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.057179 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd"] Nov 03 20:22:55.075246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.075205 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d320a902-7vctd"] Nov 03 20:22:55.089384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.089356 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:22:55.089832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.089812 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:22:55.090462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.090445 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:22:55.408291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.408259 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6c1b34-e2be-47ef-b82b-abdacb12de8f" path="/var/lib/kubelet/pods/0f6c1b34-e2be-47ef-b82b-abdacb12de8f/volumes" Nov 03 20:22:55.612075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:55.612048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.342041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.342006 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:22:56.343762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.343733 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8036b665-c559-434d-bb41-2e040e3ce501" containerName="maas-api" Nov 03 20:22:56.344088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.344069 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8036b665-c559-434d-bb41-2e040e3ce501" containerName="maas-api" Nov 03 20:22:56.344596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.344573 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8036b665-c559-434d-bb41-2e040e3ce501" containerName="maas-api" Nov 03 20:22:56.352718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.352695 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.366375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.366344 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5ca474-serving-cert\"" Nov 03 20:22:56.371484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.370582 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:22:56.399163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.399020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.490145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.480630 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.490145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.480714 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.490145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.480986 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.490145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.481053 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvmr\" (UniqueName: \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.513473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.513441 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.583121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.583065 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.584522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.584016 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvmr\" (UniqueName: \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.585074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.584973 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.586134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.586098 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.602198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.586022 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.602198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.600855 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.603658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.603555 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.606655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.606617 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvmr\" (UniqueName: \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\") pod \"maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:56.646509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.646471 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.681196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:56.681154 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:57.082182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:57.078982 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:22:57.551928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:57.551892 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.311425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.311389 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" event={"ID":"ff143bfd-2b06-48c0-8a32-0062ce724b4d","Type":"ContainerStarted","Data":"2b1791a34d0121b3bbe1bda9896817f8b9992a069c30e8986633f2930b38dad1"} Nov 03 20:22:58.311624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.311481 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" event={"ID":"ff143bfd-2b06-48c0-8a32-0062ce724b4d","Type":"ContainerStarted","Data":"1ea774342f2026489b84e239896d4a9d38e8e1f166222d5c0325ed5d67ded32a"} Nov 03 20:22:58.312360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.312337 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:22:58.335202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.335139 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" podStartSLOduration=1.665824965 podStartE2EDuration="2.335087858s" totalImagesPullingTime="669.262893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:58.33247312 +0000 UTC m=+1519.057480713" watchObservedRunningTime="2026-11-03 20:22:58.335087858 +0000 UTC m=+1519.060095457" Nov 03 20:22:58.524709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.524673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.633121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.633079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.777061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.777022 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.857928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.857135 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:22:58.857928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.857396 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" containerID="cri-o://96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896" gracePeriod=30 Nov 03 20:22:58.897309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:22:58.897232 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.193031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:00.192998 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.307726 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:00.307689 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:23:01.083087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.083056 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:01.843610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.843539 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:23:01.848218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.848197 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:01.856919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.856900 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5de824-serving-cert\"" Nov 03 20:23:01.860609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.860586 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:23:01.952543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.952503 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:01.983294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.983264 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:01.983481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.983438 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbr7\" (UniqueName: \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:01.983578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.983561 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:01.983657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:01.983640 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.070429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.070374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.099180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.092301 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.099180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.092468 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbr7\" (UniqueName: \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.099180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.092580 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.099180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.092646 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.099180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.093548 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.115132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.114366 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.128130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.125843 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.128130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.126614 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbr7\" (UniqueName: \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\") pod \"maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.166130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.165916 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:02.175374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.175202 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.394129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.393382 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.463826 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:23:02.463798 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8114fe2_e575_4777_89bb_9f2e1d373c68.slice/crio-fd921f76a15d79aade9dfc58fc99288b399424d77ca1eb7801e590f1ab209e19 WatchSource:0}: Error finding container fd921f76a15d79aade9dfc58fc99288b399424d77ca1eb7801e590f1ab209e19: Status 404 returned error can't find the container with id fd921f76a15d79aade9dfc58fc99288b399424d77ca1eb7801e590f1ab209e19 Nov 03 20:23:02.465552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.465528 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:23:02.507351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.507307 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.966528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:02.966487 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:03.431971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:03.431909 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" event={"ID":"a8114fe2-e575-4777-89bb-9f2e1d373c68","Type":"ContainerStarted","Data":"56182d0627d3b9405a9385c831f0ae45a2d00486c7bcf9f6607e79d8878b7c6d"} Nov 03 20:23:03.432257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:03.432235 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" event={"ID":"a8114fe2-e575-4777-89bb-9f2e1d373c68","Type":"ContainerStarted","Data":"fd921f76a15d79aade9dfc58fc99288b399424d77ca1eb7801e590f1ab209e19"} Nov 03 20:23:03.432397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:03.432385 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:03.434433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:03.434387 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" podStartSLOduration=1.9149398720000002 podStartE2EDuration="2.434373856s" totalImagesPullingTime="519.433984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:03.430490209 +0000 UTC m=+1524.155497809" watchObservedRunningTime="2026-11-03 20:23:03.434373856 +0000 UTC m=+1524.159381451" Nov 03 20:23:03.896565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:03.896537 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.235895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.235872 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:23:04.298973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.298939 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.301267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301241 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301296 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301395 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301480 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:04.301527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301490 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301535 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301596 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:04.301641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301619 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301656 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301702 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.301803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301705 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:04.301803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301753 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\") pod \"194691fd-dabc-47ef-90cc-94e4f303d5d6\" (UID: \"194691fd-dabc-47ef-90cc-94e4f303d5d6\") " Nov 03 20:23:04.302010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.301943 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:04.302189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302165 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:04.302604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302582 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/194691fd-dabc-47ef-90cc-94e4f303d5d6-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.302677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302613 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.302677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302629 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.302677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302643 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.302677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.302657 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.303878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.303857 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:04.304191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.304174 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:04.304349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.304333 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:04.304617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.304590 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775" pod "194691fd-dabc-47ef-90cc-94e4f303d5d6" (UID: "194691fd-dabc-47ef-90cc-94e4f303d5d6"). InnerVolumeSpecName "kube-api-access-66775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:04.413787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.407029 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66775\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-kube-api-access-66775\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.413787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.407065 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.413787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.407082 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.413787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.407096 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/194691fd-dabc-47ef-90cc-94e4f303d5d6-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:04.425978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.425937 5954 generic.go:356] "Generic (PLEG): container finished" podID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerID="96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896" exitCode=0 Nov 03 20:23:04.426192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.425998 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" Nov 03 20:23:04.426192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.426023 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerDied","Data":"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896"} Nov 03 20:23:04.426192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.426085 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9" event={"ID":"194691fd-dabc-47ef-90cc-94e4f303d5d6","Type":"ContainerDied","Data":"0b87284a03a50d71615399a6e8aa839e087594a0cfa9a0eaadbf626164ea45eb"} Nov 03 20:23:04.426192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.426143 5954 scope.go:122] "RemoveContainer" containerID="96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896" Nov 03 20:23:04.439391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.439371 5954 scope.go:122] "RemoveContainer" containerID="4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16" Nov 03 20:23:04.449800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.449778 5954 scope.go:122] "RemoveContainer" containerID="96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896" Nov 03 20:23:04.450101 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:04.450083 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896\": container with ID starting with 96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896 not found: ID does not exist" containerID="96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896" Nov 03 20:23:04.450211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.450143 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896"} err="failed to get container status \"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896\": rpc error: code = NotFound desc = could not find container \"96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896\": container with ID starting with 96e57dbb6099936f207b8977f6242c4e1d54c1c4140193ab963fa2f7ea861896 not found: ID does not exist" Nov 03 20:23:04.450211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.450162 5954 scope.go:122] "RemoveContainer" containerID="4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16" Nov 03 20:23:04.450449 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:04.450434 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16\": container with ID starting with 4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16 not found: ID does not exist" containerID="4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16" Nov 03 20:23:04.450489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.450456 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16"} err="failed to get container status \"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16\": rpc error: code = NotFound desc = could not find container \"4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16\": container with ID starting with 4bbf70274d090a1b9c54e4c0f661cbf7003699b970e9e02475434d325ea7ff16 not found: ID does not exist" Nov 03 20:23:04.456554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.456528 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:23:04.457588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.457569 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2d5927-openshift-default-6fc7674cf-gqsx9"] Nov 03 20:23:04.679121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:04.679080 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:05.335463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:05.335435 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:23:05.405007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:05.404976 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:05.412752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:05.412718 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" path="/var/lib/kubelet/pods/194691fd-dabc-47ef-90cc-94e4f303d5d6/volumes" Nov 03 20:23:06.188744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:06.188709 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.874628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:06.874590 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:07.686085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:07.686044 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:08.384330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:08.384297 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.186566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.186530 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.600997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.600970 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:23:09.613591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.613557 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:23:09.672711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.672678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.776468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.776433 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.908224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:09.908139 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.445491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:10.445206 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:23:10.908763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:10.908723 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:11.544903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:11.544869 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:23:11.545133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:11.545092 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" podUID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" containerName="maas-api" containerID="cri-o://7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d" gracePeriod=30 Nov 03 20:23:11.607410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:11.607384 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:23:11.705280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:11.705240 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.323200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.323160 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.333888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.333537 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:23:12.439143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.438548 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.451746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.451696 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\" (UniqueName: \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\") pod \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " Nov 03 20:23:12.452513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.452488 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\") pod \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " Nov 03 20:23:12.468178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.466338 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca" pod "d407f0c7-cf54-41c2-ab04-f2d71dff3394" (UID: "d407f0c7-cf54-41c2-ab04-f2d71dff3394"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:12.468178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.467012 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\") pod \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " Nov 03 20:23:12.481136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.471390 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z" pod "d407f0c7-cf54-41c2-ab04-f2d71dff3394" (UID: "d407f0c7-cf54-41c2-ab04-f2d71dff3394"). InnerVolumeSpecName "kube-api-access-b792z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:12.481136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.474178 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") pod \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\" (UID: \"d407f0c7-cf54-41c2-ab04-f2d71dff3394\") " Nov 03 20:23:12.505169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.499270 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs" pod "d407f0c7-cf54-41c2-ab04-f2d71dff3394" (UID: "d407f0c7-cf54-41c2-ab04-f2d71dff3394"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:12.509128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.506087 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls" pod "d407f0c7-cf54-41c2-ab04-f2d71dff3394" (UID: "d407f0c7-cf54-41c2-ab04-f2d71dff3394"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:12.511035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.511004 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:12.515130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.511216 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b792z\" (UniqueName: \"kubernetes.io/projected/d407f0c7-cf54-41c2-ab04-f2d71dff3394-kube-api-access-b792z\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:12.515346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.515327 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d407f0c7-cf54-41c2-ab04-f2d71dff3394-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:12.515452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.515440 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d407f0c7-cf54-41c2-ab04-f2d71dff3394-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:12.570338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.569722 5954 generic.go:356] "Generic (PLEG): container finished" podID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" containerID="7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d" exitCode=0 Nov 03 20:23:12.570338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.569782 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" Nov 03 20:23:12.570338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.569850 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" event={"ID":"d407f0c7-cf54-41c2-ab04-f2d71dff3394","Type":"ContainerDied","Data":"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d"} Nov 03 20:23:12.570338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.569917 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq" event={"ID":"d407f0c7-cf54-41c2-ab04-f2d71dff3394","Type":"ContainerDied","Data":"b38a02ddad69f60b33599add9f93f335ed0a58c4a68f9a31b257d33c267c6ac9"} Nov 03 20:23:12.570338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.570015 5954 scope.go:122] "RemoveContainer" containerID="7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d" Nov 03 20:23:12.579175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.579146 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.587702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.587682 5954 scope.go:122] "RemoveContainer" containerID="7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d" Nov 03 20:23:12.588022 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:12.588003 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d\": container with ID starting with 7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d not found: ID does not exist" containerID="7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d" Nov 03 20:23:12.588089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.588032 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d"} err="failed to get container status \"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d\": rpc error: code = NotFound desc = could not find container \"7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d\": container with ID starting with 7e695329fa4f0cc9bf7fdce6698c2e5002a1af0107d79afe9c3514c8e7b1469d not found: ID does not exist" Nov 03 20:23:12.606320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.606289 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:23:12.615594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.615568 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-15e02e6b-58bb7676cb-jr8tq"] Nov 03 20:23:12.961130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:12.960988 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:13.494137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:13.493415 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" path="/var/lib/kubelet/pods/d407f0c7-cf54-41c2-ab04-f2d71dff3394/volumes" Nov 03 20:23:14.046261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:14.046220 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:14.585910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:14.585843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:14.602035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:14.601952 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" podUID="9aad01dc-a017-43fd-8c23-b328055098a5" containerName="payload-processing" containerID="cri-o://464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130" gracePeriod=25 Nov 03 20:23:14.622354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:14.617365 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" podUID="73d1eacb-df49-4907-aec0-4468bf287c0a" containerName="payload-pre-processing" containerID="cri-o://dfa07f226cb38929d12365eb07215d837d6c993ef202fa1ecb46ce6136fcbf84" gracePeriod=25 Nov 03 20:23:14.944693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:14.944650 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:15.081156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:15.073914 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:15.579021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:15.568558 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.304558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.304510 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:16.357130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.352376 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:16.404470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.403785 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:16.408567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.408091 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:23:16.553133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.535711 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:23:16.744127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.586070 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:16.775767 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:16.775723 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.374s" Nov 03 20:23:16.775939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.775892 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777037 5954 cpu_manager.go:414] "containerMap: removing container" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777060 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777139 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" containerName="maas-api" Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777149 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" containerName="maas-api" Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777359 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:16.781135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.777382 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d407f0c7-cf54-41c2-ab04-f2d71dff3394" containerName="maas-api" Nov 03 20:23:16.888176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.887858 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.965403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.963337 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:23:16.965403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.963476 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:16.972418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.972382 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:23:16.982220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:16.978852 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-34e844-openshift-default-dockercfg-k295l\"" Nov 03 20:23:17.112132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.109297 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.112726 5954 generic.go:356] "Generic (PLEG): container finished" podID="73d1eacb-df49-4907-aec0-4468bf287c0a" containerID="dfa07f226cb38929d12365eb07215d837d6c993ef202fa1ecb46ce6136fcbf84" exitCode=0 Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.112875 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" event={"ID":"73d1eacb-df49-4907-aec0-4468bf287c0a","Type":"ContainerDied","Data":"dfa07f226cb38929d12365eb07215d837d6c993ef202fa1ecb46ce6136fcbf84"} Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.117853 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\") pod \"9aad01dc-a017-43fd-8c23-b328055098a5\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118035 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\" (UniqueName: \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\") pod \"9aad01dc-a017-43fd-8c23-b328055098a5\" (UID: \"9aad01dc-a017-43fd-8c23-b328055098a5\") " Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118352 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118524 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118597 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118625 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118669 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118750 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.118954 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2gx\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.119002 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.119063 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.137447 5954 generic.go:356] "Generic (PLEG): container finished" podID="9aad01dc-a017-43fd-8c23-b328055098a5" containerID="464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130" exitCode=0 Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.137629 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" event={"ID":"9aad01dc-a017-43fd-8c23-b328055098a5","Type":"ContainerDied","Data":"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130"} Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.137680 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" event={"ID":"9aad01dc-a017-43fd-8c23-b328055098a5","Type":"ContainerDied","Data":"171b51e57f0df6b6b2f1405ff1428f22c7ed0e8e15af8467054ceab9d06ca340"} Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.137714 5954 scope.go:122] "RemoveContainer" containerID="464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130" Nov 03 20:23:17.141192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.137874 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p" Nov 03 20:23:17.209139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.152532 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume" pod "9aad01dc-a017-43fd-8c23-b328055098a5" (UID: "9aad01dc-a017-43fd-8c23-b328055098a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:17.209139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.205707 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd" pod "9aad01dc-a017-43fd-8c23-b328055098a5" (UID: "9aad01dc-a017-43fd-8c23-b328055098a5"). InnerVolumeSpecName "kube-api-access-d2rbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:17.210848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.210765 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263039 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263228 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263306 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263332 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263377 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263451 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263646 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2gx\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263692 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263753 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263886 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aad01dc-a017-43fd-8c23-b328055098a5-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:17.281131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.263905 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2rbd\" (UniqueName: \"kubernetes.io/projected/9aad01dc-a017-43fd-8c23-b328055098a5-kube-api-access-d2rbd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:17.302133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.292252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.302133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.292568 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.302133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.292822 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.302133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.296451 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.302133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.296710 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.311010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.310538 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.360759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.360688 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.399286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.394146 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\" (UniqueName: \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\") pod \"73d1eacb-df49-4907-aec0-4468bf287c0a\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " Nov 03 20:23:17.399286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.394241 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\") pod \"73d1eacb-df49-4907-aec0-4468bf287c0a\" (UID: \"73d1eacb-df49-4907-aec0-4468bf287c0a\") " Nov 03 20:23:17.401041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.399834 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.401041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.400120 5954 scope.go:122] "RemoveContainer" containerID="464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130" Nov 03 20:23:17.403242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.401518 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2gx\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-67mzv\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.403242 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:17.401950 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130\": container with ID starting with 464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130 not found: ID does not exist" containerID="464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130" Nov 03 20:23:17.403242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.401991 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130"} err="failed to get container status \"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130\": rpc error: code = NotFound desc = could not find container \"464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130\": container with ID starting with 464b20b81bacf8371cd1495f9d9821e164aa60e152353614ef65c29d795f8130 not found: ID does not exist" Nov 03 20:23:17.406842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.405164 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume" pod "73d1eacb-df49-4907-aec0-4468bf287c0a" (UID: "73d1eacb-df49-4907-aec0-4468bf287c0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:17.409836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.409798 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz" pod "73d1eacb-df49-4907-aec0-4468bf287c0a" (UID: "73d1eacb-df49-4907-aec0-4468bf287c0a"). InnerVolumeSpecName "kube-api-access-jh7gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:17.504225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.502064 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7gz\" (UniqueName: \"kubernetes.io/projected/73d1eacb-df49-4907-aec0-4468bf287c0a-kube-api-access-jh7gz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:17.504225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.502101 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d1eacb-df49-4907-aec0-4468bf287c0a-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:17.608004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.607393 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:17.609061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.609009 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.671472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.661298 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672141 5954 cpu_manager.go:414] "containerMap: removing container" podUID="73d1eacb-df49-4907-aec0-4468bf287c0a" containerName="payload-pre-processing" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672170 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d1eacb-df49-4907-aec0-4468bf287c0a" containerName="payload-pre-processing" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672222 5954 cpu_manager.go:414] "containerMap: removing container" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672230 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672253 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9aad01dc-a017-43fd-8c23-b328055098a5" containerName="payload-processing" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672261 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aad01dc-a017-43fd-8c23-b328055098a5" containerName="payload-processing" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672461 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="194691fd-dabc-47ef-90cc-94e4f303d5d6" containerName="istio-proxy" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672476 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aad01dc-a017-43fd-8c23-b328055098a5" containerName="payload-processing" Nov 03 20:23:17.674764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.672497 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d1eacb-df49-4907-aec0-4468bf287c0a" containerName="payload-pre-processing" Nov 03 20:23:17.849506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.848377 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.918177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.902144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.937175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.937035 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.962235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.937096 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.962619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.962542 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.963023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.962973 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.963245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.963183 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.963585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.963484 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr58\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.963585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.963537 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.969171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.963917 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:17.969454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:17.969405 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.027329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.027294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:18.041143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.037827 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:23:18.067150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.067098 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f3ae23-openshift-default-dockercfg-48chf\"" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097272 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr58\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097325 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097436 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097504 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097736 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097774 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097821 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.097992 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.167939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.098022 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.210218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.175937 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.210218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.176994 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.210218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.177302 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.210218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.177545 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.421132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.398552 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr58\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.421132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.399356 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.491177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.487787 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.491177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.490989 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.540132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.532068 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:18.606549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.606512 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" event={"ID":"73d1eacb-df49-4907-aec0-4468bf287c0a","Type":"ContainerDied","Data":"cae091c271c9713b9c02a8034653bd34a3d36035635372d87fd3f6fe0621a2af"} Nov 03 20:23:18.606903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.606883 5954 scope.go:122] "RemoveContainer" containerID="dfa07f226cb38929d12365eb07215d837d6c993ef202fa1ecb46ce6136fcbf84" Nov 03 20:23:18.607163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.606823 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w" Nov 03 20:23:18.635804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.635772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:18.664454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.664419 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\") pod \"e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.861191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.861152 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:23:18.969726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.944393 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:18.997499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.983197 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:18.997499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.983487 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:18.997499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.983543 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:18.997499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:18.994161 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e50c04-688cc7bdcd-c779p"] Nov 03 20:23:19.012427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:19.012392 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:23:19.098132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:19.086200 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:23:19.098132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:19.096285 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e50c04-6d585576dc-6xp6w"] Nov 03 20:23:19.700853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:19.700815 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:20.324423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.324362 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:20.405639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.405590 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:20.406049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.406017 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:20.407928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.365192 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:23:20.408525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.366527 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:20.409142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.395905 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:20.409273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.326626 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:20.409538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.339084 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:23:20.410587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.408228 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:20.412152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.412117 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d1eacb-df49-4907-aec0-4468bf287c0a" path="/var/lib/kubelet/pods/73d1eacb-df49-4907-aec0-4468bf287c0a/volumes" Nov 03 20:23:20.413051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.413026 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aad01dc-a017-43fd-8c23-b328055098a5" path="/var/lib/kubelet/pods/9aad01dc-a017-43fd-8c23-b328055098a5/volumes" Nov 03 20:23:20.431129 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:20.413583 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.006s" Nov 03 20:23:20.434251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.434220 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:23:20.434508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.434483 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerStarted","Data":"d49745f50191252b15b5f891cb4d2299aab46a770d3a6f7d56bde0edc29517b1"} Nov 03 20:23:20.434774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.426253 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podStartSLOduration=5.426233448 podStartE2EDuration="5.426233448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:20.424733726 +0000 UTC m=+1541.149741323" watchObservedRunningTime="2026-11-03 20:23:20.426233448 +0000 UTC m=+1541.151241061" Nov 03 20:23:20.620351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.620221 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:20.621230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.621179 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:20.632047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.631247 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:23:20.632047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:20.631312 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:23:21.378146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.377825 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:21.431994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.430347 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerStarted","Data":"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1"} Nov 03 20:23:21.439289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.439243 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerStarted","Data":"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367"} Nov 03 20:23:21.439595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.439554 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerStarted","Data":"c881ae32a2cfb18c113bd40406ef9b7551030b03cf9cf1398a2ad348d840ead2"} Nov 03 20:23:21.496311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.496233 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podStartSLOduration=4.496212544 podStartE2EDuration="4.496212544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:21.477823002 +0000 UTC m=+1542.202830599" watchObservedRunningTime="2026-11-03 20:23:21.496212544 +0000 UTC m=+1542.221220140" Nov 03 20:23:21.713533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.713420 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:21.959572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.959541 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:21.965523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:21.965451 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:22.284360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.284274 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:22.452170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.451780 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:22.452170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.451828 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:22.452614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.452511 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:23:22.452811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.452790 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:23:22.797084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.797042 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:22.900421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:22.900385 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:23.583565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:23.574322 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:24.189581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:24.189518 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:24.299200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:24.299156 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:24.748744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:24.748707 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:25.524450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:25.524413 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.463953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:26.463827 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.891277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:26.891239 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:27.292524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:27.292440 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:27.402996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:27.402959 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:27.820332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:27.820238 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:27.928889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:27.926522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:28.335129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.334316 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:23:28.335129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.334411 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:28.421256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.421195 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:28.643167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.643133 5954 generic.go:356] "Generic (PLEG): container finished" podID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerID="c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615" exitCode=0 Nov 03 20:23:28.643346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.643212 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerDied","Data":"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615"} Nov 03 20:23:28.643758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.643737 5954 scope.go:122] "RemoveContainer" containerID="c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615" Nov 03 20:23:28.644995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.644976 5954 generic.go:356] "Generic (PLEG): container finished" podID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerID="b70efb7085501d9437bfe3dd1ef78114c4a58bc9ccf3affdd3365eca107ae7d4" exitCode=0 Nov 03 20:23:28.645077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.645056 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerDied","Data":"b70efb7085501d9437bfe3dd1ef78114c4a58bc9ccf3affdd3365eca107ae7d4"} Nov 03 20:23:28.645602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.645584 5954 scope.go:122] "RemoveContainer" containerID="b70efb7085501d9437bfe3dd1ef78114c4a58bc9ccf3affdd3365eca107ae7d4" Nov 03 20:23:28.646245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.646215 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.646290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.646280 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.646334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.646310 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.647207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.647183 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.647281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.647243 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.647323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:28.647280 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:29.403202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:29.403168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:29.684897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:29.684420 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerStarted","Data":"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c"} Nov 03 20:23:29.686768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:29.686736 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerStarted","Data":"1f9f022160388a824b414f4387da16e246b1e1a227c64036219652c4a391ca87"} Nov 03 20:23:30.106433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.106403 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:23:30.107575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.107547 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:30.107702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.107600 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:30.295238 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:30.295182 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:23:30.295754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.295701 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:30.359179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.358451 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:23:30.359179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.358567 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:30.359179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:30.358622 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:31.106656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.106623 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:31.107167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.106681 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:31.350128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.349586 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:31.350128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.349666 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:31.376402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.376310 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:31.966133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:31.965177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.075530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.075489 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.106435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.106403 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:32.106656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.106634 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:32.161129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.160988 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:23:32.350398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.350304 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:32.350659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.350629 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:32.480533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.480495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.602070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:32.602031 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.106655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.106433 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:33.106655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.106478 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:33.281503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.281469 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:23:33.303409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.303368 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:23:33.303779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.303624 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.376172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.375486 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:33.376172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.375561 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:33.496965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.496930 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497026 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497117 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497150 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497205 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497248 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497288 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497357 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.497440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.497388 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zhr\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.576566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.574515 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.599247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599209 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599335 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599431 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599477 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599551 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599748 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.599916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.599896 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600047 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600196 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zhr\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600416 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600457 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600090 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.600770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.600714 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.602774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.602752 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.603493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.603468 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.604341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.604313 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.683891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.683762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.695238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.687905 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zhr\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:33.708188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.707794 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.961264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.940325 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:34.057762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:33.990617 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:34.171051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.171018 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:34.172026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.171980 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:34.267310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.266337 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:34.359835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.358007 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:34.359835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.358066 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:34.451130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:34.448853 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:35.179262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.178684 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:35.179262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.178744 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:35.226801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.190368 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:35.286899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.286572 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:35.364223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.337347 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:35.390593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.385829 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:35.390593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.385904 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:35.404763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.404722 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:35.411804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.411765 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:35.412062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.412037 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:35.412294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.412272 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:35.479758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.479710 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:35.676322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:35.669301 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:23:36.095705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.084569 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.120261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.120223 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Nov 03 20:23:36.120530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.120504 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Nov 03 20:23:36.266834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.266787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.302423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.302384 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" event={"ID":"40759711-5d37-4e80-80af-938f65ca226c","Type":"ContainerStarted","Data":"a35c4c9d7a95f3127b52329d2f150718f247875e9035de30e03c1f15581911ad"} Nov 03 20:23:36.302617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.302452 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" event={"ID":"40759711-5d37-4e80-80af-938f65ca226c","Type":"ContainerStarted","Data":"3a05976d4fa4dc259c5bf6fa5a89299be53cb8169a572158c091df6221695155"} Nov 03 20:23:36.368453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.367634 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 20:23:36.368453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.367699 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 20:23:36.450505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.436885 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podStartSLOduration=3.436852438 podStartE2EDuration="3.436852438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:36.33045373 +0000 UTC m=+1557.055461338" watchObservedRunningTime="2026-11-03 20:23:36.436852438 +0000 UTC m=+1557.161860036" Nov 03 20:23:36.477130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.438852 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.991882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.991834 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:36.993224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.993197 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:36.993345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:36.993259 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:37.111880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.111850 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:23:37.113505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.113481 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:23:37.229169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.229042 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.500765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.500737 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:23:37.501847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.501804 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:23:37.503615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.503571 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:23:37.995129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.993852 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:37.995129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:37.993916 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:38.110390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:38.110344 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.931844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:38.931799 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.994130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:38.991266 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:38.994130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:38.991327 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:39.686770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:39.686737 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:39.992059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:39.991981 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:39.992059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:39.992042 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:40.493336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:40.493298 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:40.991897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:40.991865 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:40.992067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:40.991919 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:41.405583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:41.405534 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:41.991283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:41.991251 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:41.991435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:41.991306 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:42.058331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.058300 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:23:42.058500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.058481 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" podUID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" containerName="maas-api" containerID="cri-o://dfee5a99158f8f740e2af2a0be4334649d15a47ed14c32319911a144f74e4813" gracePeriod=30 Nov 03 20:23:42.068937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.068847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.267610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.267423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.499953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.499921 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.836225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.835407 5954 generic.go:356] "Generic (PLEG): container finished" podID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" containerID="dfee5a99158f8f740e2af2a0be4334649d15a47ed14c32319911a144f74e4813" exitCode=0 Nov 03 20:23:42.836225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.835635 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" event={"ID":"e6fd72e2-20fd-4682-a18a-b96f3b327e41","Type":"ContainerDied","Data":"dfee5a99158f8f740e2af2a0be4334649d15a47ed14c32319911a144f74e4813"} Nov 03 20:23:42.941761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.941720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.016190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.996038 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:43.016190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:42.996133 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 03 20:23:43.119203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.119169 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:23:43.200628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.200587 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.254597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.254556 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") pod \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " Nov 03 20:23:43.254825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.254806 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\") pod \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " Nov 03 20:23:43.254916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.254874 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\") pod \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " Nov 03 20:23:43.255169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.255145 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\" (UniqueName: \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\") pod \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\" (UID: \"e6fd72e2-20fd-4682-a18a-b96f3b327e41\") " Nov 03 20:23:43.255315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.255285 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca" pod "e6fd72e2-20fd-4682-a18a-b96f3b327e41" (UID: "e6fd72e2-20fd-4682-a18a-b96f3b327e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:43.256024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.256000 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fd72e2-20fd-4682-a18a-b96f3b327e41-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:43.257268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.257242 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls" pod "e6fd72e2-20fd-4682-a18a-b96f3b327e41" (UID: "e6fd72e2-20fd-4682-a18a-b96f3b327e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:43.257592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.257569 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7" pod "e6fd72e2-20fd-4682-a18a-b96f3b327e41" (UID: "e6fd72e2-20fd-4682-a18a-b96f3b327e41"). InnerVolumeSpecName "kube-api-access-ggjq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:43.257670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.257602 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs" pod "e6fd72e2-20fd-4682-a18a-b96f3b327e41" (UID: "e6fd72e2-20fd-4682-a18a-b96f3b327e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:43.357799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.357764 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:43.357799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.357791 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjq7\" (UniqueName: \"kubernetes.io/projected/e6fd72e2-20fd-4682-a18a-b96f3b327e41-kube-api-access-ggjq7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:43.357799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.357802 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6fd72e2-20fd-4682-a18a-b96f3b327e41-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:43.757166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.757081 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:23:43.758035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.758017 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" containerName="maas-api" Nov 03 20:23:43.758092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.758038 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" containerName="maas-api" Nov 03 20:23:43.758193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.758181 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" containerName="maas-api" Nov 03 20:23:43.763053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.763029 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.789094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.789070 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-34e844-serving-cert\"" Nov 03 20:23:43.790674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.790649 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:23:43.844205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.844099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.844478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.844461 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" Nov 03 20:23:43.844551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.844490 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc" event={"ID":"e6fd72e2-20fd-4682-a18a-b96f3b327e41","Type":"ContainerDied","Data":"19216e352ebb27616ec2f2fc705d2dbe9e87a8c6bbfe1a187f0f8e9b81e2d99d"} Nov 03 20:23:43.844551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.844545 5954 scope.go:122] "RemoveContainer" containerID="dfee5a99158f8f740e2af2a0be4334649d15a47ed14c32319911a144f74e4813" Nov 03 20:23:43.866875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.866850 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wxq\" (UniqueName: \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.867240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.867221 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.867340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.867325 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.867392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.867380 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.897402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.897364 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:23:43.906473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.906446 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e50c04-77c66dd47c-ztmmc"] Nov 03 20:23:43.962068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.961974 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.982952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.982383 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wxq\" (UniqueName: \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.982952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.982843 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.982952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.982965 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.983371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.983006 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:43.984062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:43.984036 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:44.007128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.005769 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wxq\" (UniqueName: \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:44.019177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.017798 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:44.048127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.037919 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:44.072132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.063858 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:44.072132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.065604 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:44.072132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.070340 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:23:44.086138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.086094 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:44.496886 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:23:44.496857 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840416a8_b1bd_4230_83a4_d6960dbfb0d4.slice/crio-1824a48f34c152c63826f2aecb516543b75956668cec7b88e3ac43dbca9e04e1 WatchSource:0}: Error finding container 1824a48f34c152c63826f2aecb516543b75956668cec7b88e3ac43dbca9e04e1: Status 404 returned error can't find the container with id 1824a48f34c152c63826f2aecb516543b75956668cec7b88e3ac43dbca9e04e1 Nov 03 20:23:44.497302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.497280 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:23:44.855040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.854996 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" event={"ID":"840416a8-b1bd-4230-83a4-d6960dbfb0d4","Type":"ContainerStarted","Data":"1824a48f34c152c63826f2aecb516543b75956668cec7b88e3ac43dbca9e04e1"} Nov 03 20:23:44.969455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:44.969423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:45.428476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:45.428426 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fd72e2-20fd-4682-a18a-b96f3b327e41" path="/var/lib/kubelet/pods/e6fd72e2-20fd-4682-a18a-b96f3b327e41/volumes" Nov 03 20:23:45.866473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:45.866435 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" event={"ID":"840416a8-b1bd-4230-83a4-d6960dbfb0d4","Type":"ContainerStarted","Data":"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96"} Nov 03 20:23:45.867001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:45.866535 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:45.891014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:45.890925 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" podStartSLOduration=2.346103565 podStartE2EDuration="2.890911026s" totalImagesPullingTime="544.807461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:45.88803328 +0000 UTC m=+1566.613040876" watchObservedRunningTime="2026-11-03 20:23:45.890911026 +0000 UTC m=+1566.615918619" Nov 03 20:23:46.089358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:46.088714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.153844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:47.153808 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.990683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:47.990643 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.241517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.241421 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.289984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.289946 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:23:48.297320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.297294 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.318795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.318329 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:23:48.406794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.406762 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.406990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.406915 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407078 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407210 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407290 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntwr\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407431 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407487 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407600 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.407661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.407645 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.509756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.509676 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.509940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.509764 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.509940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.509864 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.509940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.509909 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.509981 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntwr\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510037 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510065 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510223 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510318 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510618 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510696 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510699 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510815 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.510896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.510865 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.512421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.512395 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.512679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.512654 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.522295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.522271 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntwr\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.523750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.523727 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.623321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.623282 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:48.920754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.920720 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:23:48.921279 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:23:48.921246 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924f1b64_cee0_449e_98c3_1e3bca0b3b47.slice/crio-4861e9d4f2a5ec396c13431bf7e1bebd7dad3b0af3f3036fbc2d8f071933bb68 WatchSource:0}: Error finding container 4861e9d4f2a5ec396c13431bf7e1bebd7dad3b0af3f3036fbc2d8f071933bb68: Status 404 returned error can't find the container with id 4861e9d4f2a5ec396c13431bf7e1bebd7dad3b0af3f3036fbc2d8f071933bb68 Nov 03 20:23:48.924261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.924226 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:48.924365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.924309 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:48.924431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.924362 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:48.959908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:48.959871 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.867628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:49.867596 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.903292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:49.903241 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" event={"ID":"924f1b64-cee0-449e-98c3-1e3bca0b3b47","Type":"ContainerStarted","Data":"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75"} Nov 03 20:23:49.903458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:49.903324 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" event={"ID":"924f1b64-cee0-449e-98c3-1e3bca0b3b47","Type":"ContainerStarted","Data":"4861e9d4f2a5ec396c13431bf7e1bebd7dad3b0af3f3036fbc2d8f071933bb68"} Nov 03 20:23:49.936181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:49.936129 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podStartSLOduration=1.9360943769999999 podStartE2EDuration="1.936094377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:49.930100459 +0000 UTC m=+1570.655108055" watchObservedRunningTime="2026-11-03 20:23:49.936094377 +0000 UTC m=+1570.661101969" Nov 03 20:23:50.630130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:50.625471 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:50.639580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:50.639527 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:50.639775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:50.639629 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:50.987088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:50.986994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.064246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.064176 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99"] Nov 03 20:23:51.095829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.095792 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5b6be4b-zvn99"] Nov 03 20:23:51.412567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.412537 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270f7bc4-6cfe-477b-b353-f0a95ab82b5b" path="/var/lib/kubelet/pods/270f7bc4-6cfe-477b-b353-f0a95ab82b5b/volumes" Nov 03 20:23:51.464591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.464558 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.570128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.568081 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.634944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.624663 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:51.634944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.624743 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:51.883522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.883480 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:23:51.940739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.913792 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:23:51.940739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.914038 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" containerID="cri-o://2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e" gracePeriod=30 Nov 03 20:23:51.957882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:51.957845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:52.556332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:52.556222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:52.646473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:52.646424 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:52.646764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:52.646726 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:52.667452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:52.667423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:53.035070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.034514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:53.097614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.097589 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:23:53.104123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.104072 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.112214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.112195 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7ae0f1-openshift-default-dockercfg-xwqqn\"" Nov 03 20:23:53.160894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.158826 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:53.160894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.160027 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439266 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439351 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439484 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439527 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439556 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439647 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zz5\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439721 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439795 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.445130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.439840 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.665904 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.665975 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666361 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666403 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666509 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666546 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666571 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.676892 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.677192 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.678743 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.678992 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.680490 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:53.684133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.680547 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:53.687954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.687910 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.691651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.691601 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.694807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.694757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zz5\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.697333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.697269 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.706092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.666655 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zz5\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.706092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.702851 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.706092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.703734 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:53.771978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:53.771745 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:54.091404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.091356 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.091926 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:23:54.091896 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d7cb77_77fe_4bf2_bef6_b0374b2f17cd.slice/crio-8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2 WatchSource:0}: Error finding container 8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2: Status 404 returned error can't find the container with id 8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2 Nov 03 20:23:54.093926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.093902 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:23:54.095238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.095207 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:54.095304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.095288 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:54.095370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.095336 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:54.411765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.411669 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.603142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.600322 5954 patch_prober.go:29] interesting pod/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:23:54.603142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.600388 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:23:54.646570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.639219 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:54.646570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:54.639280 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:55.039647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.038226 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerStarted","Data":"5eeae0ef3b0b7fb0c72daef973c5db9234b8fd61096a76ecf042bc2f06c05a33"} Nov 03 20:23:55.039647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.038299 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerStarted","Data":"8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2"} Nov 03 20:23:55.066221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.066171 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podStartSLOduration=2.066156389 podStartE2EDuration="2.066156389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:55.06399948 +0000 UTC m=+1575.789007077" watchObservedRunningTime="2026-11-03 20:23:55.066156389 +0000 UTC m=+1575.791163984" Nov 03 20:23:55.146132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.146072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:55.641131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.640613 5954 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:23:55.641131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.640684 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 20:23:55.772610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.772574 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:55.779221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:55.779196 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:56.035186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.034573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.049294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.049265 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:56.050628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.050600 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:23:56.326146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.325337 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:23:56.342598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.342568 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.345041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.345010 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:23:56.355594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.355557 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d9f082-serving-cert\"" Nov 03 20:23:56.369942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.368100 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.414418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.413778 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g7q\" (UniqueName: \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.414418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.413845 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.414418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.414154 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.414418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.414376 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.472364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.472332 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.516152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.516074 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.516717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.516693 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.517334 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:56.516933 5954 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-serving-cert: secret "maas-api-e2e-inf-b-d9f082-serving-cert" not found Nov 03 20:23:56.517334 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:56.517009 5954 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls podName:0a32eabb-ea40-4eb5-ad0a-51c82bf34045 nodeName:}" failed. No retries permitted until 2026-11-03 20:23:57.01698816 +0000 UTC m=+1577.741995751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls") pod "maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" (UID: "0a32eabb-ea40-4eb5-ad0a-51c82bf34045") : secret "maas-api-e2e-inf-b-d9f082-serving-cert" not found Nov 03 20:23:56.517334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.516937 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g7q\" (UniqueName: \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.517334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.517176 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.518099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.518070 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.519619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.519590 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.530780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.530751 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g7q\" (UniqueName: \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:56.672651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.672612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.726249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.726216 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:56.726878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.726819 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:56.728475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:56.728449 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:23:57.027525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.027488 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:57.030534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.030506 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") pod \"maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:57.283320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.282972 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:57.653590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.653393 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759438 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759565 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759625 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759728 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759771 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759810 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.759837 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.760002 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.763627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.760150 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\") pod \"47d87097-b6a2-47fb-ae91-f38462d26dab\" (UID: \"47d87097-b6a2-47fb-ae91-f38462d26dab\") " Nov 03 20:23:57.772589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.771921 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:57.774721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.772987 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:57.774721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.772810 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:57.775036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.775007 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:57.776877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.776848 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:57.780930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.780905 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:57.782629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.782608 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:57.782726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.782680 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "kube-api-access-dxnbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:57.783473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.783450 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy" pod "47d87097-b6a2-47fb-ae91-f38462d26dab" (UID: "47d87097-b6a2-47fb-ae91-f38462d26dab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:57.838893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.838857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862121 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862152 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862169 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862182 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862196 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862212 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862228 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnbd\" (UniqueName: \"kubernetes.io/projected/47d87097-b6a2-47fb-ae91-f38462d26dab-kube-api-access-dxnbd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862243 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d87097-b6a2-47fb-ae91-f38462d26dab-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:57.862767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:57.862258 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d87097-b6a2-47fb-ae91-f38462d26dab-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:58.158678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.156925 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:23:58.239850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.239816 5954 generic.go:356] "Generic (PLEG): container finished" podID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerID="2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e" exitCode=0 Nov 03 20:23:58.246088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.245385 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" Nov 03 20:23:58.269531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.250438 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerDied","Data":"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e"} Nov 03 20:23:58.269797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.269770 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc" event={"ID":"47d87097-b6a2-47fb-ae91-f38462d26dab","Type":"ContainerDied","Data":"222e9b2a10c52e1a2ea2147af55d490982f12f30b399099181c0517da75fbf96"} Nov 03 20:23:58.269929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.269917 5954 scope.go:122] "RemoveContainer" containerID="2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e" Nov 03 20:23:58.335868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.332469 5954 scope.go:122] "RemoveContainer" containerID="5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e" Nov 03 20:23:58.389254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.389231 5954 scope.go:122] "RemoveContainer" containerID="2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e" Nov 03 20:23:58.389840 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:58.389812 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e\": container with ID starting with 2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e not found: ID does not exist" containerID="2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e" Nov 03 20:23:58.389936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.389852 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e"} err="failed to get container status \"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e\": rpc error: code = NotFound desc = could not find container \"2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e\": container with ID starting with 2fbd2fbece0fd7933a570f05e94be61e791aecfa34073c491b8579cc644fae1e not found: ID does not exist" Nov 03 20:23:58.389936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.389879 5954 scope.go:122] "RemoveContainer" containerID="5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e" Nov 03 20:23:58.390493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.390151 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:23:58.390791 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:23:58.390769 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e\": container with ID starting with 5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e not found: ID does not exist" containerID="5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e" Nov 03 20:23:58.390918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.390896 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e"} err="failed to get container status \"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e\": rpc error: code = NotFound desc = could not find container \"5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e\": container with ID starting with 5ad77ebc0360383ae60d4c55e75dddb737c97da995b6ada7c37804c8f91e412e not found: ID does not exist" Nov 03 20:23:58.394819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:58.394796 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e50c04-openshift-default-85845f7b89-fjdpc"] Nov 03 20:23:59.131730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.131502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:59.250912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.250874 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" event={"ID":"0a32eabb-ea40-4eb5-ad0a-51c82bf34045","Type":"ContainerStarted","Data":"027a57666120dde7ebf1884de40921574c9647b5881561587cd3f4444b132036"} Nov 03 20:23:59.251072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.250934 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" event={"ID":"0a32eabb-ea40-4eb5-ad0a-51c82bf34045","Type":"ContainerStarted","Data":"c709d08c3356dcab178831b4d981db6c6816235ec695528d9fbc946bc3c2a139"} Nov 03 20:23:59.251072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.251068 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:23:59.271948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.271902 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" podStartSLOduration=2.786241062 podStartE2EDuration="3.271887123s" totalImagesPullingTime="485.646061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:59.270465454 +0000 UTC m=+1579.995473049" watchObservedRunningTime="2026-11-03 20:23:59.271887123 +0000 UTC m=+1579.996894719" Nov 03 20:23:59.450186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.446904 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" path="/var/lib/kubelet/pods/47d87097-b6a2-47fb-ae91-f38462d26dab/volumes" Nov 03 20:23:59.882795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:23:59.882759 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:24:00.055131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.036707 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.172994 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174485 5954 cpu_manager.go:414] "containerMap: removing container" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174507 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174519 5954 cpu_manager.go:414] "containerMap: removing container" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174526 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174791 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.174805 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d87097-b6a2-47fb-ae91-f38462d26dab" containerName="istio-proxy" Nov 03 20:24:00.246849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.240294 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.282635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.273438 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f3ae23-serving-cert\"" Nov 03 20:24:00.282635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.278362 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.405080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.367093 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:24:00.526192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.524940 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.526192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.525076 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.551481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.544801 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.551481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.545240 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822tr\" (UniqueName: \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.567308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.564960 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.592163 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:24:00.588350 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:24:00.682131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.680474 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.682131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.680564 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.682131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.681059 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822tr\" (UniqueName: \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.682453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.682294 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.700132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.698838 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.714013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.709694 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.714013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.712643 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822tr\" (UniqueName: \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.743133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.742758 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\") pod \"maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:00.898613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:00.898533 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:01.211191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:01.199750 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:01.259188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:01.243972 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:24:01.750762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:01.750676 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" event={"ID":"d8063362-be75-459d-b3eb-b2793de04ade","Type":"ContainerStarted","Data":"f68646b3e424208e367dab47d66b12d2fc07e2aed6781cd1a7ee1b28415c360c"} Nov 03 20:24:02.235849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:02.235739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:02.397502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:02.392778 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:02.796128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:02.796007 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" event={"ID":"d8063362-be75-459d-b3eb-b2793de04ade","Type":"ContainerStarted","Data":"c67ff1c6c93fce44c585dd037118c0d1f8d7907d46a48803ea4882e9b9eb9bfb"} Nov 03 20:24:02.797091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:02.797069 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:02.857162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:02.855248 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" podStartSLOduration=3.141964253 podStartE2EDuration="3.855226905s" totalImagesPullingTime="713.262652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:02.825949139 +0000 UTC m=+1583.550956736" watchObservedRunningTime="2026-11-03 20:24:02.855226905 +0000 UTC m=+1583.580234502" Nov 03 20:24:03.179160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:03.157409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.234545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.233942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.455134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.447301 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.575205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.568993 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.670492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.670458 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.930149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.927181 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:24:04.930149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:04.927453 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" containerID="cri-o://d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9" gracePeriod=30 Nov 03 20:24:05.238138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.223836 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.340134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.333350 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:24:05.340134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.333616 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" containerID="cri-o://7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75" gracePeriod=30 Nov 03 20:24:05.562183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.557462 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.674361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.663759 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:24:05.679133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.674958 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" containerID="cri-o://a35c4c9d7a95f3127b52329d2f150718f247875e9035de30e03c1f15581911ad" gracePeriod=30 Nov 03 20:24:05.785129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:05.776050 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:24:06.094910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:06.094885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:06.209584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:06.209541 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:06.783128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:06.779593 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:07.550131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:07.549838 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.330818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:08.322699 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.811981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:08.811953 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:24:09.188585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:09.188547 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:09.879832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:09.879648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:10.509701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.509632 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:24:10.611890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.611841 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.612250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.612225 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.612812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.612793 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.613074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.613056 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.613287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.613271 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.613459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.613443 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.613698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.613678 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.612731 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614034 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614088 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\") pod \"9945dcb3-d093-4f50-bcf3-465c5682c05b\" (UID: \"9945dcb3-d093-4f50-bcf3-465c5682c05b\") " Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614276 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614338 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614706 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614980 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9945dcb3-d093-4f50-bcf3-465c5682c05b-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.614996 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.615007 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.615016 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.618142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.615245 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.620202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.620172 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.620746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.620707 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.637572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.637519 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "kube-api-access-544pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.641230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.637920 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo" pod "9945dcb3-d093-4f50-bcf3-465c5682c05b" (UID: "9945dcb3-d093-4f50-bcf3-465c5682c05b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.677657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.677632 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:24:10.715907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.715873 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.715919 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.715964 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716082 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716149 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716269 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716273 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.716360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716289 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.716360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716339 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716377 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.716630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716486 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\") pod \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\" (UID: \"924f1b64-cee0-449e-98c3-1e3bca0b3b47\") " Nov 03 20:24:10.717046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716778 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.717046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.716990 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.717235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717186 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.717414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717395 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717418 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717434 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717448 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717462 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717475 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717489 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717504 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717518 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9945dcb3-d093-4f50-bcf3-465c5682c05b-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.717880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.717534 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544pl\" (UniqueName: \"kubernetes.io/projected/9945dcb3-d093-4f50-bcf3-465c5682c05b-kube-api-access-544pl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.718697 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.718674 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.718825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.718709 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.718894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.718826 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "kube-api-access-5ntwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.719121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.719078 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo" pod "924f1b64-cee0-449e-98c3-1e3bca0b3b47" (UID: "924f1b64-cee0-449e-98c3-1e3bca0b3b47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.817402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.817369 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:10.819661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.819633 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.819768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.819669 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.819768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.819684 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntwr\" (UniqueName: \"kubernetes.io/projected/924f1b64-cee0-449e-98c3-1e3bca0b3b47-kube-api-access-5ntwr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.819768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.819701 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924f1b64-cee0-449e-98c3-1e3bca0b3b47-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.978692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:10.978612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:11.122652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.122615 5954 generic.go:356] "Generic (PLEG): container finished" podID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerID="d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9" exitCode=0 Nov 03 20:24:11.122849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.122749 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerDied","Data":"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9"} Nov 03 20:24:11.122849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.122844 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" event={"ID":"9945dcb3-d093-4f50-bcf3-465c5682c05b","Type":"ContainerDied","Data":"7f048978868e6e3870e6ace76cb564e4cc67945b0e12c78caa75bbf1ffe6879a"} Nov 03 20:24:11.123040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.122877 5954 scope.go:122] "RemoveContainer" containerID="d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9" Nov 03 20:24:11.123145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.123049 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82" Nov 03 20:24:11.139872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.139833 5954 generic.go:356] "Generic (PLEG): container finished" podID="40759711-5d37-4e80-80af-938f65ca226c" containerID="a35c4c9d7a95f3127b52329d2f150718f247875e9035de30e03c1f15581911ad" exitCode=0 Nov 03 20:24:11.140088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.140065 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" event={"ID":"40759711-5d37-4e80-80af-938f65ca226c","Type":"ContainerDied","Data":"a35c4c9d7a95f3127b52329d2f150718f247875e9035de30e03c1f15581911ad"} Nov 03 20:24:11.143157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.142527 5954 generic.go:356] "Generic (PLEG): container finished" podID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerID="7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75" exitCode=0 Nov 03 20:24:11.143157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.142634 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" event={"ID":"924f1b64-cee0-449e-98c3-1e3bca0b3b47","Type":"ContainerDied","Data":"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75"} Nov 03 20:24:11.143157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.142678 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" event={"ID":"924f1b64-cee0-449e-98c3-1e3bca0b3b47","Type":"ContainerDied","Data":"4861e9d4f2a5ec396c13431bf7e1bebd7dad3b0af3f3036fbc2d8f071933bb68"} Nov 03 20:24:11.143157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.142755 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q" Nov 03 20:24:11.166623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.166590 5954 scope.go:122] "RemoveContainer" containerID="1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503" Nov 03 20:24:11.188791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.187936 5954 scope.go:122] "RemoveContainer" containerID="d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9" Nov 03 20:24:11.190025 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:24:11.189916 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9\": container with ID starting with d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9 not found: ID does not exist" containerID="d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9" Nov 03 20:24:11.190025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.189977 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9"} err="failed to get container status \"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9\": rpc error: code = NotFound desc = could not find container \"d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9\": container with ID starting with d7574658d5fa15de4914623c1020f8353428435b6191a0fe5b115c7d20771ca9 not found: ID does not exist" Nov 03 20:24:11.190339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.190004 5954 scope.go:122] "RemoveContainer" containerID="1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503" Nov 03 20:24:11.194734 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:24:11.194518 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503\": container with ID starting with 1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503 not found: ID does not exist" containerID="1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503" Nov 03 20:24:11.194734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.194564 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503"} err="failed to get container status \"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503\": rpc error: code = NotFound desc = could not find container \"1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503\": container with ID starting with 1a810bcb795bb2e7432eeec57ff89f593e6ff88844eaddaa8a5dd32648a22503 not found: ID does not exist" Nov 03 20:24:11.194734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.194588 5954 scope.go:122] "RemoveContainer" containerID="7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75" Nov 03 20:24:11.194734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.194687 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:24:11.195568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.195467 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:24:11.197822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.197792 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-j947q"] Nov 03 20:24:11.210767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.210697 5954 scope.go:122] "RemoveContainer" containerID="7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75" Nov 03 20:24:11.211301 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:24:11.211189 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75\": container with ID starting with 7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75 not found: ID does not exist" containerID="7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75" Nov 03 20:24:11.211301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.211232 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75"} err="failed to get container status \"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75\": rpc error: code = NotFound desc = could not find container \"7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75\": container with ID starting with 7548d9da9c48321532f62123cd1ce0b093104ab9ff45f399c8433f32d42c3d75 not found: ID does not exist" Nov 03 20:24:11.216864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.216800 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:24:11.223602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.223573 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-sjx82"] Nov 03 20:24:11.235687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.235650 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.235855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.235708 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.235934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.235862 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.237170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.235963 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.237358 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.237459 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.237512 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.237557 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.237656 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\") pod \"40759711-5d37-4e80-80af-938f65ca226c\" (UID: \"40759711-5d37-4e80-80af-938f65ca226c\") " Nov 03 20:24:11.238476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.238158 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:11.238804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.238752 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:11.238991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.238971 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:11.239588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.239564 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.239686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.239590 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.239686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.239604 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.239797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.239747 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:11.242490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.242422 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:11.242490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.242451 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:11.245563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.245535 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:11.247581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.247554 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "kube-api-access-x8zhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:11.247691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.247599 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token" pod "40759711-5d37-4e80-80af-938f65ca226c" (UID: "40759711-5d37-4e80-80af-938f65ca226c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.341913 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.341953 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.341971 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40759711-5d37-4e80-80af-938f65ca226c-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.341987 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40759711-5d37-4e80-80af-938f65ca226c-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.342002 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zhr\" (UniqueName: \"kubernetes.io/projected/40759711-5d37-4e80-80af-938f65ca226c-kube-api-access-x8zhr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.342027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.342018 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40759711-5d37-4e80-80af-938f65ca226c-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.400404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.400366 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:24:11.401625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401593 5954 cpu_manager.go:414] "containerMap: removing container" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" Nov 03 20:24:11.401625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401624 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401659 5954 cpu_manager.go:414] "containerMap: removing container" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401669 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401685 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401693 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401753 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.401833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401760 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.402151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401962 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="40759711-5d37-4e80-80af-938f65ca226c" containerName="istio-proxy" Nov 03 20:24:11.402151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401979 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" containerName="istio-proxy" Nov 03 20:24:11.402151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.401989 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.402151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.402005 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" containerName="istio-proxy" Nov 03 20:24:11.436984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.436943 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.442911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.442864 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c684d-openshift-default-dockercfg-vqwnd\"" Nov 03 20:24:11.447299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.447214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:11.455335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.455305 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924f1b64-cee0-449e-98c3-1e3bca0b3b47" path="/var/lib/kubelet/pods/924f1b64-cee0-449e-98c3-1e3bca0b3b47/volumes" Nov 03 20:24:11.456410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.456386 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9945dcb3-d093-4f50-bcf3-465c5682c05b" path="/var/lib/kubelet/pods/9945dcb3-d093-4f50-bcf3-465c5682c05b/volumes" Nov 03 20:24:11.457525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.457503 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:24:11.545405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.545353 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.545578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.545421 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.545619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.545601 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.545829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.545675 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lnt\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.545829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.545753 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.546206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.546185 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.546307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.546226 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.546307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.546249 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.546406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.546387 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.556517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.556485 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661378 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661555 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661593 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661725 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661791 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lnt\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661870 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661954 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.661983 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.662008 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.663347 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.663602 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.669132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.664992 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.692537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.674467 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.692537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.691583 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.692537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.691960 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lnt\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.694585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.694214 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:24:11.694585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.694283 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:11.719953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.719871 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.721222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.720669 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.721222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.721178 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\") pod \"e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.781616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.780469 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:11.996985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:11.996803 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.259177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.258049 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" event={"ID":"40759711-5d37-4e80-80af-938f65ca226c","Type":"ContainerDied","Data":"3a05976d4fa4dc259c5bf6fa5a89299be53cb8169a572158c091df6221695155"} Nov 03 20:24:12.259177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.258156 5954 scope.go:122] "RemoveContainer" containerID="a35c4c9d7a95f3127b52329d2f150718f247875e9035de30e03c1f15581911ad" Nov 03 20:24:12.259177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.258324 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb" Nov 03 20:24:12.260449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.260425 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:24:12.271785 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:24:12.271755 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6761899d_1c05_4ecb_8373_27c9c5813ac0.slice/crio-760c3fba06abbf6e85f8c34c018c2eadf957f849aefa032b9dc08139ce240996 WatchSource:0}: Error finding container 760c3fba06abbf6e85f8c34c018c2eadf957f849aefa032b9dc08139ce240996: Status 404 returned error can't find the container with id 760c3fba06abbf6e85f8c34c018c2eadf957f849aefa032b9dc08139ce240996 Nov 03 20:24:12.276039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.275999 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:12.276176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.276084 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:12.276176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.276150 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:12.322099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.322059 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:24:12.327837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.327809 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-8sgnb"] Nov 03 20:24:12.825248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:12.825207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:13.284036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.283985 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerStarted","Data":"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f"} Nov 03 20:24:13.284600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.284353 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerStarted","Data":"760c3fba06abbf6e85f8c34c018c2eadf957f849aefa032b9dc08139ce240996"} Nov 03 20:24:13.311548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.311492 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podStartSLOduration=2.3114777650000002 podStartE2EDuration="2.311477765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:13.308691003 +0000 UTC m=+1594.033698591" watchObservedRunningTime="2026-11-03 20:24:13.311477765 +0000 UTC m=+1594.036485360" Nov 03 20:24:13.410704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.410667 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40759711-5d37-4e80-80af-938f65ca226c" path="/var/lib/kubelet/pods/40759711-5d37-4e80-80af-938f65ca226c/volumes" Nov 03 20:24:13.674956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.674911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:13.781126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.781078 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:13.783252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.783215 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:13.789281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:13.789253 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:14.299819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:14.299792 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:14.300816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:14.300793 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:24:15.022256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.022222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.500394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.500361 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:24:15.506722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.506691 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.528861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.524486 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:24:15.578550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578473 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.578696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578598 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.578806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578707 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.578806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578749 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qjb\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.578930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578874 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.578988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.578959 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.579068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.579045 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.579166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.579071 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.579166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.579097 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681337 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681444 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681562 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681624 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qjb\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681705 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681767 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681835 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681862 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.681983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.681898 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.682094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.682076 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.682255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.682227 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.683646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.683608 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.683844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.683825 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.684051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.684000 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.686014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.685994 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.686168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.686148 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.708648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.708578 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.709923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.709890 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qjb\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.733817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.733789 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.835574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.835541 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.844682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.844638 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:15.943819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:15.943749 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:16.147240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.147212 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:24:16.148098 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:24:16.148072 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed6641d_b752_4010_b07e_bca3c57fe05e.slice/crio-0c063256fb8d5dd2e3cf438b81295497ab5a381aeecfde9275f2308dea9bcb6b WatchSource:0}: Error finding container 0c063256fb8d5dd2e3cf438b81295497ab5a381aeecfde9275f2308dea9bcb6b: Status 404 returned error can't find the container with id 0c063256fb8d5dd2e3cf438b81295497ab5a381aeecfde9275f2308dea9bcb6b Nov 03 20:24:16.151252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.151219 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.151340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.151287 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.151340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.151317 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.319996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.319965 5954 generic.go:356] "Generic (PLEG): container finished" podID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerID="4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7" exitCode=0 Nov 03 20:24:16.320210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.320050 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerDied","Data":"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7"} Nov 03 20:24:16.320666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.320636 5954 scope.go:122] "RemoveContainer" containerID="4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7" Nov 03 20:24:16.322631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.322382 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" event={"ID":"9ed6641d-b752-4010-b07e-bca3c57fe05e","Type":"ContainerStarted","Data":"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a"} Nov 03 20:24:16.322631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.322439 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" event={"ID":"9ed6641d-b752-4010-b07e-bca3c57fe05e","Type":"ContainerStarted","Data":"0c063256fb8d5dd2e3cf438b81295497ab5a381aeecfde9275f2308dea9bcb6b"} Nov 03 20:24:16.323057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.323031 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.323166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.323148 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.323215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.323200 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.325892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.325813 5954 generic.go:356] "Generic (PLEG): container finished" podID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerID="58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8" exitCode=0 Nov 03 20:24:16.325892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.325837 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerDied","Data":"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8"} Nov 03 20:24:16.326428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.326404 5954 scope.go:122] "RemoveContainer" containerID="58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8" Nov 03 20:24:16.328576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.328552 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.328704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.328627 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.328784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.328709 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.432668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.431984 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podStartSLOduration=1.431964706 podStartE2EDuration="1.431964706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:16.394655118 +0000 UTC m=+1597.119662739" watchObservedRunningTime="2026-11-03 20:24:16.431964706 +0000 UTC m=+1597.156972303" Nov 03 20:24:16.845335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.845250 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:16.995868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:16.995836 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.128975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.128943 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.268338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.268299 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.351562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.351524 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerStarted","Data":"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4"} Nov 03 20:24:17.354784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.354749 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerStarted","Data":"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d"} Nov 03 20:24:17.390348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.383144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.602842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.602812 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:24:17.608472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.608437 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:17.609683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.609651 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:17.684186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.683518 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:24:17.684186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.683636 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:17.684186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.683691 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:17.852134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.849319 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:17.852134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:17.849400 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:18.472148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.467677 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:18.606130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.600481 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:18.606130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.600552 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:18.715445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.710171 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:18.715445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.710232 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:18.934129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.933692 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:18.934129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:18.933770 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:19.359232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.340014 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:24:19.392382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.391820 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:19.559134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.549814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:19.613557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.613463 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:19.613830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.613787 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:19.723171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.693629 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:19.723171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.693693 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:19.856760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.856231 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:19.856760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:19.856292 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:20.243242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.243197 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:24:20.404478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.404438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:20.682134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.681264 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:20.682134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.681338 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:20.707248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.683409 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:20.707248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.683461 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:20.850130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.849629 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:20.850130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:20.849722 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:21.478137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.477616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:21.602135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.600527 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:21.602135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.600588 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:21.682827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.682792 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:21.683186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.683139 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:21.871137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.870656 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:21.871137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:21.870716 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:22.479196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.478984 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:22.606297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.606250 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.606629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.606580 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:22.689138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.684048 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.712312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.692284 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:22.875511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.875472 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.875837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:22.875806 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:23.593297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.593249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.595219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.595197 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:23.595354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.595255 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:23.694137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.688500 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:23.694137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.688561 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:23.865322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.863084 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 03 20:24:23.865322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:23.863167 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 03 20:24:24.413715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.413192 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:24.601137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.598622 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 20:24:24.601137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.598690 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 20:24:24.685883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.685788 5954 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:24:24.685883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.685870 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 20:24:24.859446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.859415 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:24.860543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.860525 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:24.862238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:24.862215 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:24:25.553205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.553165 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:25.608497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.608464 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:24:25.687129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.681570 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:24:25.692739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.692674 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:24:25.711743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.711710 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:24:25.875999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:25.875972 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:24:26.090298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:26.090262 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:26.502940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:26.502894 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:27.600073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:27.600029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.445229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.445197 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:24:28.452768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.452722 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.461277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.461238 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7ae0f1-serving-cert\"" Nov 03 20:24:28.461730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.461708 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:24:28.490596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.490560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.587661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.587625 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.631847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.631812 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.642132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.634528 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.642790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.642686 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.643081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.643050 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6c4\" (UniqueName: \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.798152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.798048 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6c4\" (UniqueName: \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.798676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.798645 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.798807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.798758 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.798955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.798934 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.799672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.799641 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.803615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.803592 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.803768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.803669 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:28.811738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:28.811705 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6c4\" (UniqueName: \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\") pod \"maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:29.162286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:29.147989 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:29.575434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:29.575401 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:29.782707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:29.781413 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:24:29.951935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:29.951444 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" event={"ID":"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f","Type":"ContainerStarted","Data":"ef5647f85e3de0e9cbb41928bce7dfca001a59d6ffc7635b9127fe6c632fa9ca"} Nov 03 20:24:30.101987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:30.101918 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:30.465142 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:24:30.462716 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:24:30.828465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:30.828390 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:30.974131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:30.970281 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" event={"ID":"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f","Type":"ContainerStarted","Data":"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc"} Nov 03 20:24:30.974131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:30.971770 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:31.013067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:31.012998 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" podStartSLOduration=2.433688687 podStartE2EDuration="3.012975121s" totalImagesPullingTime="579.286434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:31.003744815 +0000 UTC m=+1611.728752412" watchObservedRunningTime="2026-11-03 20:24:31.012975121 +0000 UTC m=+1611.737982718" Nov 03 20:24:31.237144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:31.224669 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:32.597126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:32.596569 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:33.924133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:33.918658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.244133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:35.240753 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.274141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:35.269595 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.291209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:35.279619 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.388089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:35.388048 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.460260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:35.451749 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.213277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.211460 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.242262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.242208 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.243070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.243044 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:24:36.278242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.278199 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.524186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.418187 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.524186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.516012 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:24:36.524186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.454191 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.524186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.517784 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:24:36.540599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.449706 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:24:36.540599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.538869 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:24:36.691371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:36.655581 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:37.214285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.214236 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:24:37.214902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.214875 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:24:37.254973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.254923 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:24:37.283186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.283083 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:24:37.320412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.319423 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:24:37.323549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.323516 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:24:37.332339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.332305 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:24:37.359271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.359228 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:24:37.468216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:37.467166 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:38.076259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:38.076233 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:24:38.536129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:38.534689 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.516506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:39.516470 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.647784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:39.647746 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:40.515500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:40.515470 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.225310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:41.225276 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.327673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:41.327588 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.483212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:41.483092 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:42.302744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:42.302683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:43.297945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:43.297908 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:43.419859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:43.419827 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:43.602711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:43.602673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:44.625135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:44.580925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:45.333769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.333735 5954 generic.go:356] "Generic (PLEG): container finished" podID="610daeb7-56b0-495e-945c-92b44d93709d" containerID="3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1" exitCode=0 Nov 03 20:24:45.333959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.333797 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerDied","Data":"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1"} Nov 03 20:24:45.334433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.334413 5954 scope.go:122] "RemoveContainer" containerID="3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1" Nov 03 20:24:45.336684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.336642 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:45.336805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.336729 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:45.336805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.336760 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:45.495024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.494956 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:45.693183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:45.684094 5954 scope.go:122] "RemoveContainer" containerID="d4d4721323e00ac769e6a6e03df1294128ab589799ad8765fec0a0705dd8675a" Nov 03 20:24:46.405651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:46.404361 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerStarted","Data":"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735"} Nov 03 20:24:46.471018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:46.470983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:46.613161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:46.610570 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:24:46.613161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:46.610702 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:46.613161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:46.610760 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:47.373526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:47.373481 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:47.609442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:47.609409 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:47.609619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:47.609462 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:48.208222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:48.208191 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:48.319132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:48.311509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:48.609504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:48.609462 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:48.610007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:48.609522 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:49.166820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:49.166778 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:49.609041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:49.609005 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:49.609247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:49.609058 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:50.127632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:50.127600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:50.608632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:50.608600 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:50.608815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:50.608649 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:50.959557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:50.959453 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.365754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:51.365708 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.628190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:51.623551 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:24:51.628190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:51.623644 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 20:24:52.172501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:52.172457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:52.451714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:52.451619 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:24:52.614510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:52.614479 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:24:52.693364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:52.693335 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:24:52.930439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:52.930405 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:53.740554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:53.740517 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:54.492527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:54.492494 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.200614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:55.200582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:56.178541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:56.178508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:57.070137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:57.067714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:57.784880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:57.784845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:58.461903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:58.461870 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:59.237299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:24:59.237253 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:00.063801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.063768 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:00.270913 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:25:00.270872 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:25:00.581133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.581082 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:25:00.587664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.587624 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.622042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.622006 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:25:00.662743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.662658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:00.773037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.733925 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.773759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.773723 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.774659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.774636 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.774939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.774922 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.775169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.775150 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.775373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.775355 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.775603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.775587 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.775733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.775718 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.775896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.775870 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslw5\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.877711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877670 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.877881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877767 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.877881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877793 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.877881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877810 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.877881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877832 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.877973 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878032 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslw5\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878289 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878374 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878404 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878481 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878666 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878688 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.878822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.878807 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.880282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.880261 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.880492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.880474 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.890333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.890310 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslw5\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.890477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.890454 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.927833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.927808 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:00.966030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:00.966001 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:01.127156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.127087 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:01.226678 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:25:01.226631 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0738e36_983b_47ec_8835_e53d1358dbb0.slice/crio-e2469a5a5d80e3c712e877ec9021535d63815992b64be019bcf06e637f4bd556 WatchSource:0}: Error finding container e2469a5a5d80e3c712e877ec9021535d63815992b64be019bcf06e637f4bd556: Status 404 returned error can't find the container with id e2469a5a5d80e3c712e877ec9021535d63815992b64be019bcf06e637f4bd556 Nov 03 20:25:01.227755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.227725 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:25:01.229800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.229768 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.229897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.229832 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.229897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.229858 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.564147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.560424 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:25:01.566842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.566813 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.579665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.579636 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d0898b-openshift-default-dockercfg-2n4dg\"" Nov 03 20:25:01.582486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.582461 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:25:01.598391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.598359 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:01.629155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629092 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629171 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629204 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629299 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629484 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629558 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629672 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297n8\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629716 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.629868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.629847 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890675 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890722 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890749 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890797 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890967 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.890998 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.891098 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297n8\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.891163 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.891248 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.893156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.891810 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.904187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.902605 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.910968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.910029 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.922402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.917719 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.922402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.918218 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:01.957803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.946416 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" event={"ID":"b0738e36-983b-47ec-8835-e53d1358dbb0","Type":"ContainerStarted","Data":"6d20c4f85f833655a5ac2d7ef4a034b0ec3eb9c8270b22e146fd52d4fdcd5e7b"} Nov 03 20:25:01.957803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.946489 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" event={"ID":"b0738e36-983b-47ec-8835-e53d1358dbb0","Type":"ContainerStarted","Data":"e2469a5a5d80e3c712e877ec9021535d63815992b64be019bcf06e637f4bd556"} Nov 03 20:25:01.987141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.985838 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297n8\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:02.006802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.988258 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:02.006802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:01.991742 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:02.006802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.006318 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\") pod \"e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:02.053132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.048518 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podStartSLOduration=2.048496917 podStartE2EDuration="2.048496917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:02.005707436 +0000 UTC m=+1642.730715036" watchObservedRunningTime="2026-11-03 20:25:02.048496917 +0000 UTC m=+1642.773504514" Nov 03 20:25:02.179312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.179193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:02.195941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.195914 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:02.281144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.277800 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:02.644798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.644771 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:25:02.646304 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:25:02.646276 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7188424c_b153_4a5b_86b5_074c5560eba9.slice/crio-b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3 WatchSource:0}: Error finding container b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3: Status 404 returned error can't find the container with id b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3 Nov 03 20:25:02.649667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.649633 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:02.649816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.649707 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:02.649816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.649737 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:02.929578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.929496 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:02.938381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.938210 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:02.938953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.938655 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:02.972458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.972266 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerStarted","Data":"f79b3ee784401b302ef453a13d02ee76ee0cb532df6dde2c4be2e291e3ce2843"} Nov 03 20:25:02.972458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:02.972360 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerStarted","Data":"b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3"} Nov 03 20:25:03.006128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.001923 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podStartSLOduration=2.0019034 podStartE2EDuration="2.0019034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:02.995274058 +0000 UTC m=+1643.720281695" watchObservedRunningTime="2026-11-03 20:25:03.0019034 +0000 UTC m=+1643.726910997" Nov 03 20:25:03.142987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.142895 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:03.216194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.216144 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:03.225361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.225274 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:25:03.225668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.225636 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:25:03.929930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.929893 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:03.930251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:03.930220 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:04.091094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.091062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.237181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.225169 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:04.936130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.935537 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:04.936130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.935601 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:04.988913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.988884 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:04.989888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:04.989870 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:25:05.298347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:05.298268 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.472132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:05.460745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.570186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:05.570072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.932535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:05.932501 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:05.932855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:05.932829 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:06.400822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:06.400787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:06.844723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:06.844620 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:06.943782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:06.943748 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:06.943955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:06.943809 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:06.949802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:06.949767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:07.936158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:07.935489 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 20:25:07.936158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:07.935567 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 20:25:08.081487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:08.081387 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:09.015692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:09.015305 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:09.358004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:09.357966 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:09.359017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:09.358983 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:09.360628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:09.360604 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:25:09.779871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:09.779787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:10.511726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:10.511696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:11.309558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:11.309524 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:11.861496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:11.861447 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:11.987873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:11.987393 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.442225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.442184 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.677790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.677759 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.758521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.758443 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:25:12.764651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.764627 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.777884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.777852 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:25:12.793306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.793283 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b0adda-openshift-default-dockercfg-5rnvp\"" Nov 03 20:25:12.821859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.821809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.908442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.908400 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.909089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909048 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909256 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909290 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909348 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909496 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztj9\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909535 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909567 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.927255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.909719 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:12.930830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:12.930796 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.092671 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.092804 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.092854 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093010 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093039 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093096 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093254 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztj9\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093295 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.093614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093323 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.121130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.093820 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.121130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.095783 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.121130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.096450 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.121130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.097233 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.121130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.097496 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.199247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.155097 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.300128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.274302 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztj9\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.300128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.282777 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.300128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.288170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\") pod \"e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.416210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.391161 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:13.775856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.775819 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:13.780822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.780731 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:13.781088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.781046 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:13.781298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.781276 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:13.802805 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:25:13.802772 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe622415_b8a8_4fe5_a9a6_d796454de989.slice/crio-e1141d75ccae6b06721ffa04285063d1ef75a8dd149c89912b0b81f87a318083 WatchSource:0}: Error finding container e1141d75ccae6b06721ffa04285063d1ef75a8dd149c89912b0b81f87a318083: Status 404 returned error can't find the container with id e1141d75ccae6b06721ffa04285063d1ef75a8dd149c89912b0b81f87a318083 Nov 03 20:25:13.812633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:13.812574 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:25:14.511229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:14.511193 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerStarted","Data":"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11"} Nov 03 20:25:14.511628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:14.511254 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerStarted","Data":"e1141d75ccae6b06721ffa04285063d1ef75a8dd149c89912b0b81f87a318083"} Nov 03 20:25:14.544746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:14.544687 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podStartSLOduration=2.544674445 podStartE2EDuration="2.544674445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:14.541951249 +0000 UTC m=+1655.266958847" watchObservedRunningTime="2026-11-03 20:25:14.544674445 +0000 UTC m=+1655.269682040" Nov 03 20:25:14.568734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:14.568689 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.285717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:15.285682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.419158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:15.417942 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:15.441131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:15.439436 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:15.519491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:15.519467 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:15.520486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:15.520466 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:25:16.275537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:16.275508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:16.979121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:16.979084 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:17.086373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:17.086340 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:17.635019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:17.634977 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:18.197095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:18.197050 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:18.319279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:18.319236 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:18.740545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:18.740509 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:19.486414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:19.486383 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:19.872685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:19.872652 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:20.306787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:20.306703 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.056901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:21.056863 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.926996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:21.921812 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:22.172641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:22.172603 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:25:22.612143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:22.610916 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:23.481242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:23.481196 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:24.921198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:24.921151 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:25:25.074136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.072967 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerName="maas-api" containerID="cri-o://edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd" gracePeriod=30 Nov 03 20:25:25.105734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.099073 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:25.248092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.248052 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:25.266419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.266368 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:25.451098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.451003 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:25.874545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.874510 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:25:25.964587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.964553 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\") pod \"246e7f67-dfed-4abe-a72c-02e715aa9f48\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " Nov 03 20:25:25.965038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.964685 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\" (UniqueName: \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\") pod \"246e7f67-dfed-4abe-a72c-02e715aa9f48\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " Nov 03 20:25:25.965038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.964789 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\") pod \"246e7f67-dfed-4abe-a72c-02e715aa9f48\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " Nov 03 20:25:25.965038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.964895 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\") pod \"246e7f67-dfed-4abe-a72c-02e715aa9f48\" (UID: \"246e7f67-dfed-4abe-a72c-02e715aa9f48\") " Nov 03 20:25:25.965346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.965318 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca" pod "246e7f67-dfed-4abe-a72c-02e715aa9f48" (UID: "246e7f67-dfed-4abe-a72c-02e715aa9f48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:25.965555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.965538 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246e7f67-dfed-4abe-a72c-02e715aa9f48-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:25.966778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.966756 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs" pod "246e7f67-dfed-4abe-a72c-02e715aa9f48" (UID: "246e7f67-dfed-4abe-a72c-02e715aa9f48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:25.967226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.967209 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls" pod "246e7f67-dfed-4abe-a72c-02e715aa9f48" (UID: "246e7f67-dfed-4abe-a72c-02e715aa9f48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:25.967300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:25.967253 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f" pod "246e7f67-dfed-4abe-a72c-02e715aa9f48" (UID: "246e7f67-dfed-4abe-a72c-02e715aa9f48"). InnerVolumeSpecName "kube-api-access-97v2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:26.001331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.001291 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:26.077861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.077775 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97v2f\" (UniqueName: \"kubernetes.io/projected/246e7f67-dfed-4abe-a72c-02e715aa9f48-kube-api-access-97v2f\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.078069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.078054 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.078211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.078189 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/246e7f67-dfed-4abe-a72c-02e715aa9f48-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.270264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.270233 5954 generic.go:356] "Generic (PLEG): container finished" podID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerID="edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd" exitCode=0 Nov 03 20:25:26.270432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.270342 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" event={"ID":"246e7f67-dfed-4abe-a72c-02e715aa9f48","Type":"ContainerDied","Data":"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd"} Nov 03 20:25:26.270432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.270381 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" event={"ID":"246e7f67-dfed-4abe-a72c-02e715aa9f48","Type":"ContainerDied","Data":"6b28528f7e5cad0490e3b4b2a918dbbc06e3dd3ae5ffe2cdea47eec8bb04a9a4"} Nov 03 20:25:26.270432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.270399 5954 scope.go:122] "RemoveContainer" containerID="edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd" Nov 03 20:25:26.270548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.270518 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" Nov 03 20:25:26.284309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.284280 5954 scope.go:122] "RemoveContainer" containerID="edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd" Nov 03 20:25:26.284626 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:25:26.284602 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd\": container with ID starting with edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd not found: ID does not exist" containerID="edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd" Nov 03 20:25:26.284737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.284637 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd"} err="failed to get container status \"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd\": rpc error: code = NotFound desc = could not find container \"edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd\": container with ID starting with edb2b6bc0caa57cf17a1a022c37d2d92e7b2912d0428fff8b18851dbac1577fd not found: ID does not exist" Nov 03 20:25:26.300273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.300239 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:25:26.304344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.304320 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r"] Nov 03 20:25:26.795489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:26.795456 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:27.407883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:27.407847 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" path="/var/lib/kubelet/pods/246e7f67-dfed-4abe-a72c-02e715aa9f48/volumes" Nov 03 20:25:27.538745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:27.538704 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:28.349340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.349307 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:25:28.362194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.359491 5954 cpu_manager.go:414] "containerMap: removing container" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerName="maas-api" Nov 03 20:25:28.362194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.359522 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerName="maas-api" Nov 03 20:25:28.362194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.359781 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerName="maas-api" Nov 03 20:25:28.368775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.368748 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:25:28.369101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.369074 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.372893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.372861 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:28.375142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.375098 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d0898b-serving-cert\"" Nov 03 20:25:28.476632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.476584 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:28.479417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.479389 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2cjz\" (UniqueName: \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.479781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.479762 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.479907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.479890 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.480102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.480083 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.587130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.586003 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.587130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.586126 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.587130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.586313 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.587130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.586422 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2cjz\" (UniqueName: \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.593198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.588383 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.602476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.602393 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.618772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.618705 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.624902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.624822 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2cjz\" (UniqueName: \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\") pod \"maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:28.697131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.682491 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3ebde5-9587cd5df-j4k4r" podUID="246e7f67-dfed-4abe-a72c-02e715aa9f48" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.152:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:28.698191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:28.697913 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:29.473906 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:25:29.465274 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cacbf0a_b64e_4c51_a22b_742e2f2e6924.slice/crio-d377169263a72115c8e8d1b1d04d982c02655b620a22ed22746c91c0a930f231 WatchSource:0}: Error finding container d377169263a72115c8e8d1b1d04d982c02655b620a22ed22746c91c0a930f231: Status 404 returned error can't find the container with id d377169263a72115c8e8d1b1d04d982c02655b620a22ed22746c91c0a930f231 Nov 03 20:25:29.562707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:29.562662 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:25:29.663538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:29.663505 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.464828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:30.464787 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" event={"ID":"8cacbf0a-b64e-4c51-a22b-742e2f2e6924","Type":"ContainerStarted","Data":"d377169263a72115c8e8d1b1d04d982c02655b620a22ed22746c91c0a930f231"} Nov 03 20:25:30.490529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:30.490481 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.566812 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:25:30.566768 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:25:31.273334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:31.273301 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:31.485472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:31.483904 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" event={"ID":"8cacbf0a-b64e-4c51-a22b-742e2f2e6924","Type":"ContainerStarted","Data":"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a"} Nov 03 20:25:31.485472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:31.485358 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:31.509234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:31.509184 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" podStartSLOduration=2.929547185 podStartE2EDuration="3.50916857s" totalImagesPullingTime="579.621385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:31.505915494 +0000 UTC m=+1672.230923089" watchObservedRunningTime="2026-11-03 20:25:31.50916857 +0000 UTC m=+1672.234176166" Nov 03 20:25:31.976504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:31.976469 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.179683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.179651 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.280737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.280647 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.531582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.531441 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:25:32.531824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.531796 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" containerID="cri-o://1f9f022160388a824b414f4387da16e246b1e1a227c64036219652c4a391ca87" gracePeriod=30 Nov 03 20:25:32.541915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.541848 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.981945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:32.981856 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.260035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:34.254981 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.914061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:34.914024 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:35.782131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:35.778772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:36.493980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:36.493942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:37.114607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.114569 5954 patch_prober.go:29] interesting pod/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:25:37.115031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.114646 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:25:37.296541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.296506 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:37.626480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.626445 5954 generic.go:356] "Generic (PLEG): container finished" podID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerID="5eeae0ef3b0b7fb0c72daef973c5db9234b8fd61096a76ecf042bc2f06c05a33" exitCode=0 Nov 03 20:25:37.626646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.626515 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerDied","Data":"5eeae0ef3b0b7fb0c72daef973c5db9234b8fd61096a76ecf042bc2f06c05a33"} Nov 03 20:25:37.627044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.627027 5954 scope.go:122] "RemoveContainer" containerID="5eeae0ef3b0b7fb0c72daef973c5db9234b8fd61096a76ecf042bc2f06c05a33" Nov 03 20:25:37.628986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.628957 5954 generic.go:356] "Generic (PLEG): container finished" podID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerID="1f9f022160388a824b414f4387da16e246b1e1a227c64036219652c4a391ca87" exitCode=0 Nov 03 20:25:37.629094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.629030 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerDied","Data":"1f9f022160388a824b414f4387da16e246b1e1a227c64036219652c4a391ca87"} Nov 03 20:25:37.629094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.629076 5954 scope.go:122] "RemoveContainer" containerID="b70efb7085501d9437bfe3dd1ef78114c4a58bc9ccf3affdd3365eca107ae7d4" Nov 03 20:25:37.629473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.629440 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:37.629553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.629514 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:37.629592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:37.629557 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:38.063926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.063889 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:38.478733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.478705 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:25:38.562628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.561859 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:25:38.585067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585021 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585158 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585273 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585367 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585573 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585644 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585692 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585775 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585788 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585833 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\") pod \"493a7ba1-730d-4156-aee0-91c7865fd71a\" (UID: \"493a7ba1-730d-4156-aee0-91c7865fd71a\") " Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.585957 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586007 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586294 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586794 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586816 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493a7ba1-730d-4156-aee0-91c7865fd71a-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586833 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.586934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.586847 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.587799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.587173 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.589732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.589692 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "kube-api-access-86r2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.589967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.589935 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:38.590997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.590967 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.592746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.592722 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy" pod "493a7ba1-730d-4156-aee0-91c7865fd71a" (UID: "493a7ba1-730d-4156-aee0-91c7865fd71a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.655018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.654982 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerStarted","Data":"e0f289509ba14dac3dbe99ab2c6101345fd7376e01d619ae33a88b2e58a92197"} Nov 03 20:25:38.658036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.658007 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" event={"ID":"493a7ba1-730d-4156-aee0-91c7865fd71a","Type":"ContainerDied","Data":"d7bdd933b20404f03c1a7a4b2e9e770fbb2bec8c5e402d95b7f0c1d7b18af24e"} Nov 03 20:25:38.658248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.658054 5954 scope.go:122] "RemoveContainer" containerID="1f9f022160388a824b414f4387da16e246b1e1a227c64036219652c4a391ca87" Nov 03 20:25:38.658248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.658190 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd" Nov 03 20:25:38.688983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.688957 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.688983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.688983 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.689205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.688993 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493a7ba1-730d-4156-aee0-91c7865fd71a-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.689205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.689003 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86r2s\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-kube-api-access-86r2s\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.689205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.689015 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493a7ba1-730d-4156-aee0-91c7865fd71a-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.700723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.700663 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:25:38.737130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.720961 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3ebde5-openshift-default-64df64dfb9-rljzd"] Nov 03 20:25:38.779870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.774461 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:25:38.779870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.779101 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:38.779870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.779193 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:38.876448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:38.876413 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.417131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.413206 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" path="/var/lib/kubelet/pods/493a7ba1-730d-4156-aee0-91c7865fd71a/volumes" Nov 03 20:25:39.466306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.466267 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.578828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.578793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.684567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.684477 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.786134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.782574 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:39.786134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.782638 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:39.801134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:39.797841 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:40.096186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:40.095076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:40.808133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:40.804495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:40.820662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:40.812368 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:40.820662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:40.817719 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:40.952253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:40.952211 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:41.069190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:41.066356 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:25:41.777195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:41.777157 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:41.777415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:41.777219 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:41.866874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:41.866838 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.585893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:42.585860 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.780368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:42.776905 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:42.780368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:42.776993 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:43.339072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.339032 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:43.439069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.439018 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:43.829137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.827248 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:43.829137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.827316 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:43.838671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.838592 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:43.974151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:43.971561 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:44.104205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:44.101624 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:44.772870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:44.772831 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:25:44.773314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:44.772896 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 20:25:44.986575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:44.986532 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:45.375533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:45.375501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:45.489483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:45.489444 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:45.771188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:45.763208 5954 scope.go:122] "RemoveContainer" containerID="aedb7099d24399e109ff5d549e73737c5505dbe015b2dcffa0dcb604939a023a" Nov 03 20:25:45.817018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:45.816984 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:25:45.840438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:45.840407 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:25:46.229914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:46.229855 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.975014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:46.974953 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:47.773000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:47.772968 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:48.299681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:48.299645 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:48.512654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:48.512625 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:49.443275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:49.443210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:50.205070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:50.205040 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:51.276179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:51.272796 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:51.984061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:51.984023 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:52.806083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:52.806046 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:53.493290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:53.493245 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:53.493465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:53.493337 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:53.902141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:53.901610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:55.197518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:55.197480 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.396928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.396885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.498009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.497967 5954 generic.go:356] "Generic (PLEG): container finished" podID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerID="c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367" exitCode=0 Nov 03 20:25:56.498350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.498325 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerDied","Data":"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367"} Nov 03 20:25:56.500167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.499038 5954 scope.go:122] "RemoveContainer" containerID="c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367" Nov 03 20:25:56.519526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.519481 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.519816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.519794 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.519953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.519936 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.542692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:56.542656 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:57.360722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.360680 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:57.556135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.550007 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerStarted","Data":"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309"} Nov 03 20:25:57.851548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.851520 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:57.961430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.961397 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:25:57.961848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.961815 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:25:57.962077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:57.962048 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.165229 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166589 5954 cpu_manager.go:414] "containerMap: removing container" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166611 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166631 5954 cpu_manager.go:414] "containerMap: removing container" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166640 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166920 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.168191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.166938 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="493a7ba1-730d-4156-aee0-91c7865fd71a" containerName="istio-proxy" Nov 03 20:25:58.170516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.170490 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.179609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.179564 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d9f082-openshift-default-dockercfg-mggr7\"" Nov 03 20:25:58.186163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.186127 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:25:58.252363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252311 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzh7\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.252572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252376 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.252572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252466 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.252715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252578 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.252715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252636 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.252833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.252723 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.253025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.253007 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.253194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.253049 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.253290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.253221 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.359963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.359916 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360004 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360086 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360388 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360435 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360469 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360501 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzh7\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360540 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.360635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.360627 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.363771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.362216 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.363771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.362497 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.363771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.363080 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.363771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.363359 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.367201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.365714 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.380899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.380878 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.384175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.384148 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzh7\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.384510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.384493 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.385675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.385647 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.553860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.553775 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:25:58.864673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.864647 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:25:58.865179 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:25:58.865152 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1cb072_9af5_4e81_9f70_c061ae37fb4c.slice/crio-3daa50c1605fa361574492d9a69474038578bbf324b61dabe65560084d399e39 WatchSource:0}: Error finding container 3daa50c1605fa361574492d9a69474038578bbf324b61dabe65560084d399e39: Status 404 returned error can't find the container with id 3daa50c1605fa361574492d9a69474038578bbf324b61dabe65560084d399e39 Nov 03 20:25:58.868687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.868656 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:58.868780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.868741 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:58.868780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.868771 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:58.959295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.959265 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:25:58.959427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:58.959326 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:25:59.078817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.078774 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:59.612135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.611489 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" event={"ID":"5d1cb072-9af5-4e81-9f70-c061ae37fb4c","Type":"ContainerStarted","Data":"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256"} Nov 03 20:25:59.612135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.611563 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" event={"ID":"5d1cb072-9af5-4e81-9f70-c061ae37fb4c","Type":"ContainerStarted","Data":"3daa50c1605fa361574492d9a69474038578bbf324b61dabe65560084d399e39"} Nov 03 20:25:59.651721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.649719 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podStartSLOduration=1.649701069 podStartE2EDuration="1.649701069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:59.647666854 +0000 UTC m=+1700.372674433" watchObservedRunningTime="2026-11-03 20:25:59.649701069 +0000 UTC m=+1700.374708666" Nov 03 20:25:59.958979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.958902 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:25:59.958979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:25:59.958952 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:26:00.089287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.089242 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:00.436642 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:00.424970 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:26:00.557128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.556085 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:26:00.557128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.556233 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:00.557128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.556279 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:00.963187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.959421 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:26:00.963187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:00.959486 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:26:01.003260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:01.003223 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:01.557131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:01.555060 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:01.557131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:01.555145 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:01.964131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:01.963208 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:26:01.964131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:01.963289 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:26:02.141216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:02.132224 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:02.555594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:02.555551 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.555892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:02.555861 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:02.960130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:02.959524 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.960130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:02.959595 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:26:03.206622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:03.206591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:03.561205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:03.557751 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.561205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:03.557813 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:03.981587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:03.981552 5954 patch_prober.go:29] interesting pod/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.981869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:03.981843 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 20:26:04.268189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:04.263391 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:04.560408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:04.556556 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:04.560408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:04.556648 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:05.009411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.009378 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:26:05.009975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.009951 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:26:05.011489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.011469 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:26:05.143531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.143482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:05.563134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.557700 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:05.563134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.557775 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:05.897790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:05.897743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:06.560506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:06.560468 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:06.560845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:06.560814 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:07.081130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:07.077680 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:07.554220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:07.554177 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Nov 03 20:26:07.554906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:07.554876 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Nov 03 20:26:07.945632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:07.945595 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:08.132672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:08.132636 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:08.561532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:08.561503 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:26:08.563199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:08.563172 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:26:08.564180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:08.564162 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:26:09.307220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.307193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:09.961122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.957288 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:26:09.964516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.964491 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:26:09.964702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.964627 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:09.973538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.973498 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c2c601-serving-cert\"" Nov 03 20:26:09.981931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:09.981903 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:10.062587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.062554 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.062801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.062662 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.062801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.062785 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.063067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.063038 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmx6\" (UniqueName: \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.078004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.077976 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:10.168139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.167732 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.168139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.167856 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.168139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.167930 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.168139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.168085 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmx6\" (UniqueName: \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.197135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.194462 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.206133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.199641 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.216744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.216655 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmx6\" (UniqueName: \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.232683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.232647 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\") pod \"maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.290504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.290466 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:10.990885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:10.990809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:11.081266 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:26:11.081196 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fc622d_4e5d_4d9e_a2db_625b225afeca.slice/crio-1f4aba128e8774bfa1a4af9d8b1a4bdb624f4447ede55d6b7616997320ec287e WatchSource:0}: Error finding container 1f4aba128e8774bfa1a4af9d8b1a4bdb624f4447ede55d6b7616997320ec287e: Status 404 returned error can't find the container with id 1f4aba128e8774bfa1a4af9d8b1a4bdb624f4447ede55d6b7616997320ec287e Nov 03 20:26:11.081713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.081685 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:26:11.083240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.083219 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:26:11.902352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.902309 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" event={"ID":"49fc622d-4e5d-4d9e-a2db-625b225afeca","Type":"ContainerStarted","Data":"e8515ad42c99bb940f620e9dfb7079eca4144f7d3a53c677447c9acd8b659e2b"} Nov 03 20:26:11.902557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.902384 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" event={"ID":"49fc622d-4e5d-4d9e-a2db-625b225afeca","Type":"ContainerStarted","Data":"1f4aba128e8774bfa1a4af9d8b1a4bdb624f4447ede55d6b7616997320ec287e"} Nov 03 20:26:11.902557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.902416 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:11.925957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.925908 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" podStartSLOduration=2.462309081 podStartE2EDuration="2.925892509s" totalImagesPullingTime="463.583428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:11.92241946 +0000 UTC m=+1712.647427059" watchObservedRunningTime="2026-11-03 20:26:11.925892509 +0000 UTC m=+1712.650900105" Nov 03 20:26:11.970222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:11.970183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:12.675636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:12.675600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:13.502120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:13.502077 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:14.104423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:14.104385 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:14.305766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:14.302900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.336687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:15.336644 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.449206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:15.440687 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:16.196353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.196324 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:16.889354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.889320 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:16.958102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.958026 5954 generic.go:356] "Generic (PLEG): container finished" podID="7188424c-b153-4a5b-86b5-074c5560eba9" containerID="f79b3ee784401b302ef453a13d02ee76ee0cb532df6dde2c4be2e291e3ce2843" exitCode=0 Nov 03 20:26:16.958263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.958094 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerDied","Data":"f79b3ee784401b302ef453a13d02ee76ee0cb532df6dde2c4be2e291e3ce2843"} Nov 03 20:26:16.958596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.958579 5954 scope.go:122] "RemoveContainer" containerID="f79b3ee784401b302ef453a13d02ee76ee0cb532df6dde2c4be2e291e3ce2843" Nov 03 20:26:16.960671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.960645 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:16.960794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.960736 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:16.960794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:16.960772 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:17.792129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:17.788793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:17.945267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:17.944261 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:26:17.972748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:17.972701 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerStarted","Data":"301e46984ef0c7e56ec73043ba458c9f34d8f6367b807d426f1c8ae6d7a2800b"} Nov 03 20:26:18.196576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.196490 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:26:18.197518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.197486 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:18.197651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.197548 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:18.459890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.459804 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:26:18.460087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.460039 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" containerID="cri-o://fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b" gracePeriod=30 Nov 03 20:26:18.678198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:18.678153 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:19.196890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:19.196854 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:19.197431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:19.196909 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:19.483570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:19.483473 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:19.989751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:19.989659 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:26:20.197102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:20.197067 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:20.197536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:20.197152 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:20.376994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:20.376961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:21.187322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:21.187286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:21.196538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:21.196504 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:21.196682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:21.196582 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:21.973957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:21.973911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:22.197337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:22.197298 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:22.197532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:22.197352 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:22.685990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:22.685953 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.196905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.196869 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:26:23.197376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.196932 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 03 20:26:23.345378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.345325 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:23.345569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.345416 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:23.517790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.517751 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.652031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:23.651958 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.059774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.059350 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:26:24.082360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.082326 5954 generic.go:356] "Generic (PLEG): container finished" podID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerID="fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b" exitCode=0 Nov 03 20:26:24.082652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.082626 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" event={"ID":"2ea8e0cd-ff84-484e-adad-c3552e6f1161","Type":"ContainerDied","Data":"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b"} Nov 03 20:26:24.082748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.082685 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" event={"ID":"2ea8e0cd-ff84-484e-adad-c3552e6f1161","Type":"ContainerDied","Data":"6da9ce9c524c010d5e94749de1bfdde513755a2aab0fa7a0caeba8dce6c3099c"} Nov 03 20:26:24.082748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.082718 5954 scope.go:122] "RemoveContainer" containerID="fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b" Nov 03 20:26:24.082918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.082901 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4" Nov 03 20:26:24.097968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.097943 5954 scope.go:122] "RemoveContainer" containerID="fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b" Nov 03 20:26:24.098348 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:24.098323 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b\": container with ID starting with fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b not found: ID does not exist" containerID="fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b" Nov 03 20:26:24.098571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.098361 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b"} err="failed to get container status \"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b\": rpc error: code = NotFound desc = could not find container \"fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b\": container with ID starting with fae85f6283527345f2d282ad5ad43ec52375aa01e1e910deda491aa554f3165b not found: ID does not exist" Nov 03 20:26:24.195488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.195389 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.195488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.195471 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.195745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.195508 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.195911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.195888 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.196332 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.196487 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.196585 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.196680 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.196784 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.197007 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\") pod \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\" (UID: \"2ea8e0cd-ff84-484e-adad-c3552e6f1161\") " Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.197520 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.197905 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:24.199055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.198601 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:24.199957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.199085 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:24.199957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.199333 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "kube-api-access-t7vb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:24.200203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.200171 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:24.204560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204538 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:26:24.204937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204913 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204944 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204960 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204974 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.204988 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.205003 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vb6\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-kube-api-access-t7vb6\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.205040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.205017 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.206532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.206497 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:26:24.234925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.234891 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:24.236601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.236575 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo" pod "2ea8e0cd-ff84-484e-adad-c3552e6f1161" (UID: "2ea8e0cd-ff84-484e-adad-c3552e6f1161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:24.307621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.307250 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.307621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.307283 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea8e0cd-ff84-484e-adad-c3552e6f1161-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.425922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.425881 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:26:24.430151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.430122 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-csnb4"] Nov 03 20:26:24.580279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:24.580244 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.327715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:25.322173 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.461609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:25.461574 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" path="/var/lib/kubelet/pods/2ea8e0cd-ff84-484e-adad-c3552e6f1161/volumes" Nov 03 20:26:26.091019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:26.090972 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:27.008678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:27.007536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:27.690180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:27.690133 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.407258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:28.407228 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.945778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:28.945745 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:26:28.946025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:28.945997 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" podUID="d8063362-be75-459d-b3eb-b2793de04ade" containerName="maas-api" containerID="cri-o://c67ff1c6c93fce44c585dd037118c0d1f8d7907d46a48803ea4882e9b9eb9bfb" gracePeriod=30 Nov 03 20:26:29.065680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.065647 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.273186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.269767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.317128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.313686 5954 generic.go:356] "Generic (PLEG): container finished" podID="d8063362-be75-459d-b3eb-b2793de04ade" containerID="c67ff1c6c93fce44c585dd037118c0d1f8d7907d46a48803ea4882e9b9eb9bfb" exitCode=0 Nov 03 20:26:29.317128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.314071 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" event={"ID":"d8063362-be75-459d-b3eb-b2793de04ade","Type":"ContainerDied","Data":"c67ff1c6c93fce44c585dd037118c0d1f8d7907d46a48803ea4882e9b9eb9bfb"} Nov 03 20:26:29.521178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.520767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.753894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.753856 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:26:29.755714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.755677 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.811819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.811784 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\") pod \"d8063362-be75-459d-b3eb-b2793de04ade\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " Nov 03 20:26:29.812066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.812052 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\") pod \"d8063362-be75-459d-b3eb-b2793de04ade\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " Nov 03 20:26:29.812218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.812129 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\" (UniqueName: \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\") pod \"d8063362-be75-459d-b3eb-b2793de04ade\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " Nov 03 20:26:29.812218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.812177 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\") pod \"d8063362-be75-459d-b3eb-b2793de04ade\" (UID: \"d8063362-be75-459d-b3eb-b2793de04ade\") " Nov 03 20:26:29.812873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.812841 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca" pod "d8063362-be75-459d-b3eb-b2793de04ade" (UID: "d8063362-be75-459d-b3eb-b2793de04ade"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:29.815685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.815643 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr" pod "d8063362-be75-459d-b3eb-b2793de04ade" (UID: "d8063362-be75-459d-b3eb-b2793de04ade"). InnerVolumeSpecName "kube-api-access-822tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:29.817472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.817448 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs" pod "d8063362-be75-459d-b3eb-b2793de04ade" (UID: "d8063362-be75-459d-b3eb-b2793de04ade"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:29.819867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.819839 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls" pod "d8063362-be75-459d-b3eb-b2793de04ade" (UID: "d8063362-be75-459d-b3eb-b2793de04ade"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:29.916158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.915563 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:29.916158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.915630 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822tr\" (UniqueName: \"kubernetes.io/projected/d8063362-be75-459d-b3eb-b2793de04ade-kube-api-access-822tr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:29.916158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.915646 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8063362-be75-459d-b3eb-b2793de04ade-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:29.916158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.915659 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8063362-be75-459d-b3eb-b2793de04ade-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:29.995234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:29.993310 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:30.261307 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:30.261218 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:26:30.323790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.323757 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" Nov 03 20:26:30.323970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.323787 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn" event={"ID":"d8063362-be75-459d-b3eb-b2793de04ade","Type":"ContainerDied","Data":"f68646b3e424208e367dab47d66b12d2fc07e2aed6781cd1a7ee1b28415c360c"} Nov 03 20:26:30.323970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.323852 5954 scope.go:122] "RemoveContainer" containerID="c67ff1c6c93fce44c585dd037118c0d1f8d7907d46a48803ea4882e9b9eb9bfb" Nov 03 20:26:30.357555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.357523 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:26:30.363644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.363616 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3ae23-67c6fb4888-x9lpn"] Nov 03 20:26:30.567691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:30.567611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:31.336767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:31.336738 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:31.458701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:31.458662 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8063362-be75-459d-b3eb-b2793de04ade" path="/var/lib/kubelet/pods/d8063362-be75-459d-b3eb-b2793de04ade/volumes" Nov 03 20:26:32.104265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:32.104231 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.789678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:32.789646 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:33.638214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:33.630399 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:34.614877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:34.614845 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:34.921165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:34.921072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.390285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:35.390252 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.507195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:35.505787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.852451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:35.852416 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.406194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.406151 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.622460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.622425 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.769184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.763294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.947544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.947513 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:26:36.947758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.947733 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" containerID="cri-o://b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309" gracePeriod=30 Nov 03 20:26:36.981446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:36.981317 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.832500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:37.832469 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.933423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:37.933393 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.231323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.231290 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:26:38.232268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232231 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" Nov 03 20:26:38.232268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232254 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" Nov 03 20:26:38.232268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232274 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d8063362-be75-459d-b3eb-b2793de04ade" containerName="maas-api" Nov 03 20:26:38.232610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232281 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8063362-be75-459d-b3eb-b2793de04ade" containerName="maas-api" Nov 03 20:26:38.232610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232545 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea8e0cd-ff84-484e-adad-c3552e6f1161" containerName="istio-proxy" Nov 03 20:26:38.232610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.232562 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8063362-be75-459d-b3eb-b2793de04ade" containerName="maas-api" Nov 03 20:26:38.237870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.237847 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.247523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.247499 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-45e569-openshift-default-dockercfg-vqb8n\"" Nov 03 20:26:38.251157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.251131 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:26:38.271980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.271890 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.314571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314533 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.314805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314679 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.314805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314747 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.314951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314816 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.314951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314870 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.315071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.314964 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmks\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.315071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.315030 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.315257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.315235 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.315336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.315267 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417210 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmks\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417293 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417384 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417498 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417615 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417680 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417748 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.417802 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.418746 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.419356 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.422128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.419602 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.443144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.442540 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.472142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.460052 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmks\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.472142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.460385 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.488173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.485680 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.514146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.513762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.589144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.584255 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\") pod \"e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:38.594695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.594503 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.716938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.715950 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.866130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:38.863469 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:39.200544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.200452 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:39.213047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.213020 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:26:39.214136 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:26:39.214091 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8840ebc_c04f_48b4_b9af_0048a52ec74e.slice/crio-3a1880f46bfe91c83433e87f19a34bfba53396d5416308323118030c319956cc WatchSource:0}: Error finding container 3a1880f46bfe91c83433e87f19a34bfba53396d5416308323118030c319956cc: Status 404 returned error can't find the container with id 3a1880f46bfe91c83433e87f19a34bfba53396d5416308323118030c319956cc Nov 03 20:26:39.217248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.217214 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:39.217342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.217279 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:39.217342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.217309 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:39.622089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.622050 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerStarted","Data":"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60"} Nov 03 20:26:39.622294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.622171 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerStarted","Data":"3a1880f46bfe91c83433e87f19a34bfba53396d5416308323118030c319956cc"} Nov 03 20:26:39.648177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.648073 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podStartSLOduration=1.648059494 podStartE2EDuration="1.648059494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:39.646161449 +0000 UTC m=+1740.371169071" watchObservedRunningTime="2026-11-03 20:26:39.648059494 +0000 UTC m=+1740.373067134" Nov 03 20:26:39.871177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.866922 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:39.942851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:39.942766 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:40.053553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:40.053505 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:40.299749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:40.299650 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:40.871628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:40.871591 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:41.264365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:41.264271 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.365714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:41.365683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.747426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:41.747379 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:41.800180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:41.798807 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:26:41.973864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:41.973830 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.152458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.152400 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.177678 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:42.177637 5954 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ecaf80_c3d8_4d49_abae_5691b97763c7.slice/crio-conmon-b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ecaf80_c3d8_4d49_abae_5691b97763c7.slice/crio-b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:26:42.282016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.281945 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:26:42.355127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355079 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355188 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355255 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355346 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355457 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355488 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355528 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355573 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355575 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:42.355765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355613 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:42.355849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355829 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\") pod \"46ecaf80-c3d8-4d49-abae-5691b97763c7\" (UID: \"46ecaf80-c3d8-4d49-abae-5691b97763c7\") " Nov 03 20:26:42.355906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.355853 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:42.356385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356008 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:42.356385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356259 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:42.356732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356713 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ecaf80-c3d8-4d49-abae-5691b97763c7-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.356825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356739 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.356825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356753 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.356825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356769 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.356825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.356782 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.357638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.357611 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:42.357795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.357778 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:42.358445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.358418 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:42.359360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.359335 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58" pod "46ecaf80-c3d8-4d49-abae-5691b97763c7" (UID: "46ecaf80-c3d8-4d49-abae-5691b97763c7"). InnerVolumeSpecName "kube-api-access-qrr58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:42.458629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.458584 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.458629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.458621 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.458629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.458636 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ecaf80-c3d8-4d49-abae-5691b97763c7-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.458884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.458650 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr58\" (UniqueName: \"kubernetes.io/projected/46ecaf80-c3d8-4d49-abae-5691b97763c7-kube-api-access-qrr58\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:42.661802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.661762 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.812836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.812800 5954 generic.go:356] "Generic (PLEG): container finished" podID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerID="b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309" exitCode=0 Nov 03 20:26:42.813028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.812870 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" Nov 03 20:26:42.813226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.813200 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerDied","Data":"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309"} Nov 03 20:26:42.813374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.813358 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg" event={"ID":"46ecaf80-c3d8-4d49-abae-5691b97763c7","Type":"ContainerDied","Data":"c881ae32a2cfb18c113bd40406ef9b7551030b03cf9cf1398a2ad348d840ead2"} Nov 03 20:26:42.813485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.813473 5954 scope.go:122] "RemoveContainer" containerID="b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309" Nov 03 20:26:42.829474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.829455 5954 scope.go:122] "RemoveContainer" containerID="c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367" Nov 03 20:26:42.840609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.840580 5954 scope.go:122] "RemoveContainer" containerID="b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309" Nov 03 20:26:42.840915 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:42.840894 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309\": container with ID starting with b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309 not found: ID does not exist" containerID="b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309" Nov 03 20:26:42.840998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.840927 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309"} err="failed to get container status \"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309\": rpc error: code = NotFound desc = could not find container \"b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309\": container with ID starting with b85c95f3c95194fba0dbc190bc3bbc52677776e0fd8a65415505ed1066a66309 not found: ID does not exist" Nov 03 20:26:42.840998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.840956 5954 scope.go:122] "RemoveContainer" containerID="c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367" Nov 03 20:26:42.841261 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:26:42.841235 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367\": container with ID starting with c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367 not found: ID does not exist" containerID="c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367" Nov 03 20:26:42.841317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.841271 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367"} err="failed to get container status \"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367\": rpc error: code = NotFound desc = could not find container \"c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367\": container with ID starting with c64f0424d0e571ea28c25c618ff2fbcf55a82fddc5c84f5db5b92a4176c8d367 not found: ID does not exist" Nov 03 20:26:42.844198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.844173 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:26:42.848341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:42.848319 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f3ae23-openshift-default-5d7fd574c5-4rmvg"] Nov 03 20:26:43.266431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:43.266396 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:43.391258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:43.391223 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:43.419216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:43.419181 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" path="/var/lib/kubelet/pods/46ecaf80-c3d8-4d49-abae-5691b97763c7/volumes" Nov 03 20:26:43.505600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:43.505553 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:44.391366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:44.391327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:45.096026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:45.095990 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:45.778893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:45.778864 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:46.523330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:46.523293 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.194511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:47.194473 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.892071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:47.892041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.352830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:48.352797 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.700157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:48.700046 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.380506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:49.380472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.960604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:49.960569 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:50.062117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:50.062081 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:50.191656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:50.191623 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:50.952297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:50.952262 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:51.748321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:51.748288 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:52.580488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:52.580455 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.272316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:53.272282 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.005244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:54.005210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.763757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:54.763712 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:55.487488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:55.487440 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:56.182921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:56.182887 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.074155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:57.074119 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.730912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:57.730878 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:58.591895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:58.591858 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:59.309128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:59.307022 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:59.901730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:26:59.901695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.260169 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:27:00.260074 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:27:00.629686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.629651 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:27:00.631230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631197 5954 cpu_manager.go:414] "containerMap: removing container" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.631230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631229 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.631426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631272 5954 cpu_manager.go:414] "containerMap: removing container" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.631426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631280 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.631562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631546 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.631617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.631572 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ecaf80-c3d8-4d49-abae-5691b97763c7" containerName="istio-proxy" Nov 03 20:27:00.641407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.641384 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.642740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.642713 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:27:00.645375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.645357 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a744eb-serving-cert\"" Nov 03 20:27:00.668010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.667962 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.698731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.698703 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.698866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.698769 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhv8\" (UniqueName: \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.698933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.698907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.699134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.699119 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.767053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.767018 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.803135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.802154 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.803135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.802295 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.803135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.802386 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhv8\" (UniqueName: \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.803135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.802548 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.804134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.803663 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.840257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.840179 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.863570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.862089 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhv8\" (UniqueName: \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.876016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.875971 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\") pod \"maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:00.958772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:00.958681 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:01.398370 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:27:01.398319 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16ae07f_74bc_467b_a290_fea50234a9c6.slice/crio-c54299b3b1ec5853173a5e5244cd221ca903ece8e24f90a7a75e3417ef791a51 WatchSource:0}: Error finding container c54299b3b1ec5853173a5e5244cd221ca903ece8e24f90a7a75e3417ef791a51: Status 404 returned error can't find the container with id c54299b3b1ec5853173a5e5244cd221ca903ece8e24f90a7a75e3417ef791a51 Nov 03 20:27:01.475372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:01.475341 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:27:01.684414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:01.684335 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.350746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:02.350652 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" event={"ID":"d16ae07f-74bc-467b-a290-fea50234a9c6","Type":"ContainerStarted","Data":"daf95d62d32e480cc966403b0d5b5da8bdd6cbad22f7252948dc7b163d47e53e"} Nov 03 20:27:02.350746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:02.350714 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:02.350746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:02.350734 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" event={"ID":"d16ae07f-74bc-467b-a290-fea50234a9c6","Type":"ContainerStarted","Data":"c54299b3b1ec5853173a5e5244cd221ca903ece8e24f90a7a75e3417ef791a51"} Nov 03 20:27:02.381325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:02.381265 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" podStartSLOduration=1.7723031630000001 podStartE2EDuration="2.381246724s" totalImagesPullingTime="608.943561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:02.380418597 +0000 UTC m=+1763.105426194" watchObservedRunningTime="2026-11-03 20:27:02.381246724 +0000 UTC m=+1763.106254325" Nov 03 20:27:02.484446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:02.484417 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.018676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.018641 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:27:03.019401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.019028 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" containerID="cri-o://a9589d5db2c486e02b9cb51c9bcb574a29a7f7a65dcdcf9a36d6af3a77d25da2" gracePeriod=30 Nov 03 20:27:03.246140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.246097 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:27:03.264904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.264777 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:27:03.264904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.264900 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.271733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.271670 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-45e569-serving-cert\"" Nov 03 20:27:03.281618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.281589 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rtc\" (UniqueName: \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.281770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.281749 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.282027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.282005 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.282078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.282065 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.310924 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.310887 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.425193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.423903 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.480603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.440214 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.480603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.440545 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rtc\" (UniqueName: \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.480603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.440775 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.480603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.441974 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.480603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.477136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.489244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.489170 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.520508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.520474 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rtc\" (UniqueName: \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.577035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.576571 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\") pod \"maas-api-e2e-inf-a-45e569-8787f9468-lkfxn\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.596172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.595866 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:03.772682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:03.772646 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:27:04.102195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:04.099483 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:27:04.216201 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:27:04.211393 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacff8dd6_d623_4be2_b6c3_4262087956ae.slice/crio-a0a675f12e0cf04dde63050924b1ee55b88552d245f4de583291841575fdbf4b WatchSource:0}: Error finding container a0a675f12e0cf04dde63050924b1ee55b88552d245f4de583291841575fdbf4b: Status 404 returned error can't find the container with id a0a675f12e0cf04dde63050924b1ee55b88552d245f4de583291841575fdbf4b Nov 03 20:27:04.367641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:04.367561 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" event={"ID":"acff8dd6-d623-4be2-b6c3-4262087956ae","Type":"ContainerStarted","Data":"a0a675f12e0cf04dde63050924b1ee55b88552d245f4de583291841575fdbf4b"} Nov 03 20:27:04.442056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:04.442009 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.377084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:05.377044 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" event={"ID":"acff8dd6-d623-4be2-b6c3-4262087956ae","Type":"ContainerStarted","Data":"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40"} Nov 03 20:27:05.377337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:05.377148 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:05.399792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:05.399758 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.401096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:05.401053 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" podStartSLOduration=1.6623093039999999 podStartE2EDuration="2.401040955s" totalImagesPullingTime="738.731651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:05.397860041 +0000 UTC m=+1766.122867638" watchObservedRunningTime="2026-11-03 20:27:05.401040955 +0000 UTC m=+1766.126048550" Nov 03 20:27:06.458908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:06.458872 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:07.301981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:07.296303 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:07.515651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:07.515611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.264140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.260197 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.344190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.343911 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:27:08.362134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.351203 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:27:08.372140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.371289 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.376136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.372495 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:27:08.446129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.445388 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:27:08.467155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.458194 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.699132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.671862 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.776131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.673233 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.920132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.907220 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.977190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.976774 5954 generic.go:356] "Generic (PLEG): container finished" podID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerID="a9589d5db2c486e02b9cb51c9bcb574a29a7f7a65dcdcf9a36d6af3a77d25da2" exitCode=0 Nov 03 20:27:08.977190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:08.976938 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" event={"ID":"399211f6-d373-45c5-abc6-ff9d6b671d3d","Type":"ContainerDied","Data":"a9589d5db2c486e02b9cb51c9bcb574a29a7f7a65dcdcf9a36d6af3a77d25da2"} Nov 03 20:27:09.194697 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.194659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.197008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.196978 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 20:27:09.197259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.197236 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 20:27:09.270190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.262134 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.345921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.303725 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:27:09.345921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.328294 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 20:27:09.572390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.572291 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:27:09.575123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.573128 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:27:09.575123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.573351 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:27:09.798858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.798805 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:27:09.940913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.940874 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:27:09.941677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.941659 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:27:09.941865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.941852 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:27:09.952201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:09.952173 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:27:10.503736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.350618 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.503736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.351161 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:27:10.503736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.357379 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:27:10.637210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.595795 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 20:27:10.726399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.662693 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.726399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.671317 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 03 20:27:10.745131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.671350 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:27:10.745131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.710809 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.883603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.865598 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:10.994253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:10.994203 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Nov 03 20:27:11.353340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.353285 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:11.368137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.361210 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 03 20:27:11.700267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.668321 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:27:11.724827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.713147 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:11.738708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.738341 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:11.738877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.738859 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:11.739560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.739024 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:27:11.741163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.740818 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.741163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.740921 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.741163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.741020 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.741163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.741066 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.741477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.741190 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.745094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.744834 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:11.747178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.746817 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" Nov 03 20:27:11.747712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.747661 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:11.807158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.780219 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "kube-api-access-q77kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:11.807158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.790447 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.807158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.791520 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.807158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.791562 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77kd\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-kube-api-access-q77kd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.807158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.791577 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.830295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.830226 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.844784 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.926180 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.926240 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.926275 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\") pod \"399211f6-d373-45c5-abc6-ff9d6b671d3d\" (UID: \"399211f6-d373-45c5-abc6-ff9d6b671d3d\") " Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.927430 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.931147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.927460 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:11.936482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.936446 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:11.959464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.959375 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:11.975160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:11.967346 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:12.084132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.072172 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399211f6-d373-45c5-abc6-ff9d6b671d3d-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:12.084132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.072213 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:12.084132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.072230 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:12.084132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.072765 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:12.242071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.241945 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:12.410439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.409806 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy" pod "399211f6-d373-45c5-abc6-ff9d6b671d3d" (UID: "399211f6-d373-45c5-abc6-ff9d6b671d3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:27:12.461655 5954 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.063s" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.461771 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.461959 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.461992 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw" event={"ID":"399211f6-d373-45c5-abc6-ff9d6b671d3d","Type":"ContainerDied","Data":"aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3"} Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462088 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462214 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462270 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462332 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462385 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462436 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462494 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:27:12.471190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.462494 5954 scope.go:122] "RemoveContainer" containerID="a9589d5db2c486e02b9cb51c9bcb574a29a7f7a65dcdcf9a36d6af3a77d25da2" Nov 03 20:27:12.612843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.609747 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399211f6-d373-45c5-abc6-ff9d6b671d3d-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:12.667071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:12.659501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:13.206282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:13.163438 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:27:13.206282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:13.163504 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-4lvqw"] Nov 03 20:27:13.225139 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:27:13.223603 5954 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399211f6_d373_45c5_abc6_ff9d6b671d3d.slice/crio-aec086f0a40a185e6afec167077a55c08c5a94c34e6e444ff39b788a8fa9ecc3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399211f6_d373_45c5_abc6_ff9d6b671d3d.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:27:13.422473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:13.413144 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" path="/var/lib/kubelet/pods/399211f6-d373-45c5-abc6-ff9d6b671d3d/volumes" Nov 03 20:27:13.496480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:13.496358 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:14.308790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:14.308758 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:14.986602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:14.986571 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:15.890514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:15.890474 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:16.194418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:16.194322 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:16.302041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:16.302002 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:16.839816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:16.839780 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:17.754930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:17.754843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:18.485587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:18.485552 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:19.199449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:19.199408 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:19.896621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:19.896582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:20.668383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:20.668295 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:21.381872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.381818 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:21.988924 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.988891 5954 generic.go:356] "Generic (PLEG): container finished" podID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerID="de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f" exitCode=0 Nov 03 20:27:21.989381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.988935 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerDied","Data":"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f"} Nov 03 20:27:21.989492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.989473 5954 scope.go:122] "RemoveContainer" containerID="de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f" Nov 03 20:27:21.991021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.990989 5954 generic.go:356] "Generic (PLEG): container finished" podID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerID="c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11" exitCode=0 Nov 03 20:27:21.991146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.991052 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerDied","Data":"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11"} Nov 03 20:27:21.991432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.991417 5954 scope.go:122] "RemoveContainer" containerID="c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11" Nov 03 20:27:21.991695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.991668 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:21.991765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.991742 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:21.991992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.991783 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:21.993227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.993201 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:21.993325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.993277 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:21.993397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:21.993328 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:22.084762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:22.083396 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:22.991617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:22.991580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:23.025128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.013703 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerStarted","Data":"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3"} Nov 03 20:27:23.057201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.053584 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerStarted","Data":"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007"} Nov 03 20:27:23.396193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.396052 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:27:23.396400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.396263 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:23.396400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.396304 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:23.781049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.780953 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:27:23.781256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.781152 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:23.781256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.781202 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:23.781838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:23.781814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.396673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:24.396633 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:24.397078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:24.396687 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:24.577647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:24.577608 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.796261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:24.792542 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:24.796261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:24.792605 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:25.278246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.278202 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:27:25.286157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.283239 5954 cpu_manager.go:414] "containerMap: removing container" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" Nov 03 20:27:25.286157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.285904 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" Nov 03 20:27:25.286403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.286233 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="399211f6-d373-45c5-abc6-ff9d6b671d3d" containerName="istio-proxy" Nov 03 20:27:25.289957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.289934 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:27:25.290123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.290046 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.295820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.295798 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c2c601-openshift-default-dockercfg-zq6pw\"" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355494 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355577 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355618 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355653 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355681 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355744 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355872 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.364131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.355907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvbp\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.371433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.371397 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.374391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.374354 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:25.396624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.396591 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:25.396775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.396662 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:25.540471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.540366 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.540751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.540725 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvbp\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.541306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.541280 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.584161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.584125 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.584486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.584463 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.584736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.584714 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.584986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.584963 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.585164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.585146 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.585515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.585493 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.586023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.568949 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.587184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.587139 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.593281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.592752 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.593281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.592848 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.628789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.628757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.654233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.645998 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvbp\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.654233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.653772 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.688095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.675216 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.702129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.701569 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:25.781571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.781534 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:25.781749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.781586 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:25.911143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:25.911087 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:26.074157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.074127 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:26.391693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.390471 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:27:26.395309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.395265 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:26.399779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.396525 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:26.399779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.396589 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:26.399779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.399638 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:26.399779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.399710 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:26.781231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.781151 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:26.781231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.781204 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:26.833130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:26.832568 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.397330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.397294 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:27.397551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.397393 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:27.436133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.413505 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" event={"ID":"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c","Type":"ContainerStarted","Data":"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9"} Nov 03 20:27:27.503175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.476203 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" event={"ID":"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c","Type":"ContainerStarted","Data":"54210336b0da5b5579abe13b9b62026f80e5d0272e4fe14023b59473be7baf3f"} Nov 03 20:27:27.632471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.621497 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.635199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.635141 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podStartSLOduration=2.635120782 podStartE2EDuration="2.635120782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:27.518887144 +0000 UTC m=+1788.243894734" watchObservedRunningTime="2026-11-03 20:27:27.635120782 +0000 UTC m=+1788.360128377" Nov 03 20:27:27.782439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.782350 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:27.782712 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.782679 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:27.933377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.933347 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:27.934693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.934664 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:27.934994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:27.934967 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:28.404130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.403860 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:28.404130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.403936 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:28.598841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.598801 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:28.789195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.788232 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:28.789195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.788309 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:28.911618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.911518 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:28.911618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:28.911574 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:29.299958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.299916 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:27:29.434130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.409471 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:29.434130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.409529 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:29.434130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.412881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:29.781210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.781164 5954 patch_prober.go:29] interesting pod/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:27:29.782298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.781854 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Nov 03 20:27:29.912194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.912153 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:29.912371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:29.912225 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:30.384144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.381640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.397047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.396553 5954 patch_prober.go:29] interesting pod/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:27:30.397047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.396618 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 03 20:27:30.521128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.520128 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:27:30.611986 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:27:30.611942 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:27:30.911802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.911762 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:30.912455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.912422 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:30.987856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.987826 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:27:30.989430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:30.989403 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:27:31.093441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.092296 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.242096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.242006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.472786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.472754 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:27:31.475148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.475079 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:27:31.912167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.912092 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:31.912594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:31.912207 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:32.071371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:32.071325 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:32.789227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:32.789192 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:32.889345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:32.889313 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:32.912075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:32.912045 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:32.912265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:32.912142 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:33.029991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.029958 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:33.531132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.530928 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:33.809699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.809617 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:33.911523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.911486 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:33.911694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.911551 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:33.957204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.957163 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:27:33.968239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.966429 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:33.968444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.966321 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:27:33.973193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.971318 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c60d2a-openshift-default-dockercfg-c4xvf\"" Nov 03 20:27:33.986391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:33.986363 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:34.079535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079453 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.079694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079575 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.079694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079658 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.079834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079817 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.079953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079940 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.079990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079972 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.080025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.079994 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wsn\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.080061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.080024 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.080185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.080166 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.107723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.107692 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.190708 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.190881 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191023 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191063 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191098 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wsn\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191159 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191230 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191361 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.191438 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.194132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.192047 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.225133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.222390 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.225133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.222651 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.225133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.223265 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.225133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.223502 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.236045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.229925 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.248131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.245349 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.272128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.268679 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.272128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.269360 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wsn\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\") pod \"e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.362873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.362451 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:34.911542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.911500 5954 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 03 20:27:34.911743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:34.911592 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 03 20:27:35.056832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.056775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:35.058652 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:27:35.058620 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6989da16_dbc6_425e_94f5_39b8f2777431.slice/crio-95c4a0fcbe87bae124204b3e676835e3d89deaf945cb44f4ede124740964b3a9 WatchSource:0}: Error finding container 95c4a0fcbe87bae124204b3e676835e3d89deaf945cb44f4ede124740964b3a9: Status 404 returned error can't find the container with id 95c4a0fcbe87bae124204b3e676835e3d89deaf945cb44f4ede124740964b3a9 Nov 03 20:27:35.059428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.059404 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:27:35.114168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.112459 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:35.114168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.112614 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:35.114168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.112661 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:35.794294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.794257 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerStarted","Data":"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7"} Nov 03 20:27:35.794482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.794306 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerStarted","Data":"95c4a0fcbe87bae124204b3e676835e3d89deaf945cb44f4ede124740964b3a9"} Nov 03 20:27:35.820100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.820045 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podStartSLOduration=2.820030253 podStartE2EDuration="2.820030253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:35.818543781 +0000 UTC m=+1796.543551414" watchObservedRunningTime="2026-11-03 20:27:35.820030253 +0000 UTC m=+1796.545037848" Nov 03 20:27:35.878671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.878635 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:35.916635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.916604 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:35.917133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.917100 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:35.917711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:35.917696 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:27:36.363689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:36.363662 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:36.370840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:36.370817 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:36.801883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:36.801805 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:36.802868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:36.802848 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:27:37.610738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:37.610705 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:37.710262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:37.710231 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:38.217635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:38.217593 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:38.328145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:38.328102 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:38.618516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:38.618482 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:39.525660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:39.525615 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:39.637747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:39.637712 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:40.624979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:40.624931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:41.606218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:41.606187 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:42.400623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:42.400573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:43.299975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:43.299939 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:44.286479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:44.286446 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:45.105241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:45.105209 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:46.044479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:46.044437 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:46.781048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:46.781008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:47.680648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:47.680607 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:48.407037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:48.407004 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:48.504441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:48.504406 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.281705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:49.281666 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.387097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:49.387064 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:50.212800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:50.212711 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:51.045648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:51.045610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:51.712948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:51.712916 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:52.468232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:52.468200 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:53.391503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:53.391448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:54.177052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:54.177018 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:55.076719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:55.076682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:55.973021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:55.972986 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:56.789023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:56.788988 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:57.502807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:57.502775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:58.195482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:58.195448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:58.900230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:58.900196 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:59.796681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:27:59.796625 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:00.261361 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:28:00.261326 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:28:00.600014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:00.599925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:00.727342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:00.727300 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:01.469278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:01.469245 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.288343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:02.288313 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.423418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:02.423378 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.516189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:02.516100 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:03.305270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:03.305014 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:04.212382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:04.212348 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:05.069326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:05.069291 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:05.777252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:05.777219 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:06.745131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:06.737709 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:07.618868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:07.618828 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:08.978219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:08.978184 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:10.139170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:10.129233 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:11.154374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:11.154341 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:11.338591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:11.338535 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.391977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:12.391943 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:13.237140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:13.237075 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.136056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:14.136016 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.838133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:14.824732 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.938199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:14.938159 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:15.088890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:15.088808 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:15.884409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:15.884371 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.708596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:16.708560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:17.580754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:17.580717 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:18.354075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:18.354040 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:19.200194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.200152 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:19.848380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.848301 5954 generic.go:356] "Generic (PLEG): container finished" podID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerID="e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60" exitCode=0 Nov 03 20:28:19.848598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.848373 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerDied","Data":"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60"} Nov 03 20:28:19.848882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.848867 5954 scope.go:122] "RemoveContainer" containerID="e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60" Nov 03 20:28:19.850977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.850946 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.851082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.851014 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.851082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:19.851045 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:20.173551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.173511 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:20.858623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.858592 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerStarted","Data":"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e"} Nov 03 20:28:20.864965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.864943 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:28:20.865181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.865163 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.865235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.865203 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:20.959168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:20.959092 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:21.703134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:21.693610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:21.793530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:21.793491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:21.865197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:21.865162 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.865608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:21.865229 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:21.893831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:21.893791 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:22.864619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:22.864582 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.864839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:22.864637 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:23.035134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.029368 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:23.659335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.659226 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:28:23.660709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.660653 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" podUID="a8114fe2-e575-4777-89bb-9f2e1d373c68" containerName="maas-api" containerID="cri-o://56182d0627d3b9405a9385c831f0ae45a2d00486c7bcf9f6607e79d8878b7c6d" gracePeriod=30 Nov 03 20:28:23.669553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.668269 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:23.871008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.870532 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.871008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.870603 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:23.944128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.943862 5954 generic.go:356] "Generic (PLEG): container finished" podID="a8114fe2-e575-4777-89bb-9f2e1d373c68" containerID="56182d0627d3b9405a9385c831f0ae45a2d00486c7bcf9f6607e79d8878b7c6d" exitCode=0 Nov 03 20:28:23.944128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:23.943975 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" event={"ID":"a8114fe2-e575-4777-89bb-9f2e1d373c68","Type":"ContainerDied","Data":"56182d0627d3b9405a9385c831f0ae45a2d00486c7bcf9f6607e79d8878b7c6d"} Nov 03 20:28:24.143797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.143740 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:24.471138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.470699 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:24.605545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.605497 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:24.605783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.605751 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:28:24.780831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.778453 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\") pod \"a8114fe2-e575-4777-89bb-9f2e1d373c68\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " Nov 03 20:28:24.780831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.778640 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\" (UniqueName: \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\") pod \"a8114fe2-e575-4777-89bb-9f2e1d373c68\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " Nov 03 20:28:24.780831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.778752 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\") pod \"a8114fe2-e575-4777-89bb-9f2e1d373c68\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " Nov 03 20:28:24.782945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.781514 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\") pod \"a8114fe2-e575-4777-89bb-9f2e1d373c68\" (UID: \"a8114fe2-e575-4777-89bb-9f2e1d373c68\") " Nov 03 20:28:24.812729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.810693 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca" pod "a8114fe2-e575-4777-89bb-9f2e1d373c68" (UID: "a8114fe2-e575-4777-89bb-9f2e1d373c68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:24.818158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.818060 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs" pod "a8114fe2-e575-4777-89bb-9f2e1d373c68" (UID: "a8114fe2-e575-4777-89bb-9f2e1d373c68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:24.869768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.869711 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7" pod "a8114fe2-e575-4777-89bb-9f2e1d373c68" (UID: "a8114fe2-e575-4777-89bb-9f2e1d373c68"). InnerVolumeSpecName "kube-api-access-8jbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:24.877043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.871710 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.877043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.871777 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:24.877043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.875229 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls" pod "a8114fe2-e575-4777-89bb-9f2e1d373c68" (UID: "a8114fe2-e575-4777-89bb-9f2e1d373c68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:24.884385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.884359 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8114fe2-e575-4777-89bb-9f2e1d373c68-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:24.884521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.884387 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:24.884521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.884400 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jbr7\" (UniqueName: \"kubernetes.io/projected/a8114fe2-e575-4777-89bb-9f2e1d373c68-kube-api-access-8jbr7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:24.884521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.884414 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8114fe2-e575-4777-89bb-9f2e1d373c68-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:24.899038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.899011 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:24.957862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.957828 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" Nov 03 20:28:24.958037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.957826 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9" event={"ID":"a8114fe2-e575-4777-89bb-9f2e1d373c68","Type":"ContainerDied","Data":"fd921f76a15d79aade9dfc58fc99288b399424d77ca1eb7801e590f1ab209e19"} Nov 03 20:28:24.958037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.957972 5954 scope.go:122] "RemoveContainer" containerID="56182d0627d3b9405a9385c831f0ae45a2d00486c7bcf9f6607e79d8878b7c6d" Nov 03 20:28:24.993596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:24.993558 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:28:25.001736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:25.001705 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5de824-7bf8cbbbb6-q7zj9"] Nov 03 20:28:25.408811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:25.408776 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8114fe2-e575-4777-89bb-9f2e1d373c68" path="/var/lib/kubelet/pods/a8114fe2-e575-4777-89bb-9f2e1d373c68/volumes" Nov 03 20:28:25.630754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:25.630711 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:25.864546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:25.864516 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.864738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:25.864575 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:26.489633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:26.489591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:26.747412 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:26.747319 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:28:26.864536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:26.864499 5954 patch_prober.go:29] interesting pod/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.864721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:26.864549 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 03 20:28:27.342733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:27.342691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:27.869262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:27.869232 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:28:27.870417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:27.870392 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:28:28.211824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:28.211730 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:29.000046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:29.000013 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:29.968929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:29.968837 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:30.262249 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:28:30.262159 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:28:31.172783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:31.172744 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.949581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:31.949542 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.050740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:32.050705 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.155700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:32.155671 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.443833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:32.443801 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:28:32.444079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:32.444054 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" containerID="cri-o://e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d" gracePeriod=30 Nov 03 20:28:32.495301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:32.495237 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:33.697867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:33.697831 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:34.315299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:34.315268 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:34.526565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:34.526530 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.563878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:35.563839 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.665274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:35.665207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:36.486956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:36.486914 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:37.223438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.223402 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:37.762078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.762046 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:28:37.796735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.796707 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.796909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.796755 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.796909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.796817 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.796917 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.796940 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797060 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797246 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.797373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797351 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.797440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797420 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797497 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.797689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797647 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\") pod \"bd43e8ac-3fd7-49df-b061-f55be5c48375\" (UID: \"bd43e8ac-3fd7-49df-b061-f55be5c48375\") " Nov 03 20:28:37.798233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.797735 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.798233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.798153 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:37.798990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.798965 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.799148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799134 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.799306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799273 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.799423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799403 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd43e8ac-3fd7-49df-b061-f55be5c48375-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.799423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799308 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:28:37.799728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799679 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.799828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799742 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:37.799828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.799771 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "kube-api-access-2ctcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:37.801196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.801169 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data" pod "bd43e8ac-3fd7-49df-b061-f55be5c48375" (UID: "bd43e8ac-3fd7-49df-b061-f55be5c48375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.900912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.900880 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.900912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.900906 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.900912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.900916 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ctcx\" (UniqueName: \"kubernetes.io/projected/bd43e8ac-3fd7-49df-b061-f55be5c48375-kube-api-access-2ctcx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.901187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.900926 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.901187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:37.900936 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd43e8ac-3fd7-49df-b061-f55be5c48375-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.153667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.153579 5954 generic.go:356] "Generic (PLEG): container finished" podID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerID="e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d" exitCode=0 Nov 03 20:28:38.153667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.153629 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" Nov 03 20:28:38.153667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.153656 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerDied","Data":"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d"} Nov 03 20:28:38.153900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.153718 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx" event={"ID":"bd43e8ac-3fd7-49df-b061-f55be5c48375","Type":"ContainerDied","Data":"8297cee85f9b125978f16d17340f09aea14b93ccd6f037e64f02c47eac058f52"} Nov 03 20:28:38.153900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.153748 5954 scope.go:122] "RemoveContainer" containerID="e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d" Nov 03 20:28:38.199136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.198585 5954 scope.go:122] "RemoveContainer" containerID="58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8" Nov 03 20:28:38.199136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.199020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:38.242018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.241978 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:28:38.254377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.254346 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-kdpqx"] Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.270713 5954 scope.go:122] "RemoveContainer" containerID="e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d" Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:28:38.271247 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d\": container with ID starting with e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d not found: ID does not exist" containerID="e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d" Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.271287 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d"} err="failed to get container status \"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d\": rpc error: code = NotFound desc = could not find container \"e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d\": container with ID starting with e994d84853a6d0898ee9a05b6ae9723208c9c8084a454f47e790e94962d59d3d not found: ID does not exist" Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.271319 5954 scope.go:122] "RemoveContainer" containerID="58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8" Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:28:38.271638 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8\": container with ID starting with 58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8 not found: ID does not exist" containerID="58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8" Nov 03 20:28:38.278320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:38.271668 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8"} err="failed to get container status \"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8\": rpc error: code = NotFound desc = could not find container \"58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8\": container with ID starting with 58ba84e75e4e058ae52f1d50a7f3cf5aa27e5f7614e489729b5d9e11332609b8 not found: ID does not exist" Nov 03 20:28:39.010592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:39.010553 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:39.409927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:39.409892 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" path="/var/lib/kubelet/pods/bd43e8ac-3fd7-49df-b061-f55be5c48375/volumes" Nov 03 20:28:39.884095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:39.884046 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:40.716406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:40.716368 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:41.480290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:41.480254 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.186745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:42.186704 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.838441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:42.838406 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:28:42.838641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:42.838616 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" containerID="cri-o://6dcc59cefe0ddaea4ffcae8f6a4fc112808177c3baa026db90918a266b8f9869" gracePeriod=30 Nov 03 20:28:42.953800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:42.953766 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.306021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:43.305982 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.414877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:43.414843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.973924 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:43.973881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:44.714871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:44.714832 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:45.593894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:45.593844 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:46.280801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:46.280770 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:47.208935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:47.208874 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.095768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.095725 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.256000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.255954 5954 generic.go:356] "Generic (PLEG): container finished" podID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerID="6dcc59cefe0ddaea4ffcae8f6a4fc112808177c3baa026db90918a266b8f9869" exitCode=0 Nov 03 20:28:48.261981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.256270 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" event={"ID":"23cbfdf8-dc95-4da3-b374-f413b7186bdb","Type":"ContainerDied","Data":"6dcc59cefe0ddaea4ffcae8f6a4fc112808177c3baa026db90918a266b8f9869"} Nov 03 20:28:48.262314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.262283 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" event={"ID":"23cbfdf8-dc95-4da3-b374-f413b7186bdb","Type":"ContainerDied","Data":"e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8"} Nov 03 20:28:48.262652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.262625 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0aae3663d823474b6e767cb72848a5c641c7299db663edb223e8925c49396d8" Nov 03 20:28:48.262780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.262574 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:28:48.410921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.410793 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.410921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.410914 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.410988 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411059 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411125 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411356 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411416 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411549 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.411625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.411584 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\") pod \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\" (UID: \"23cbfdf8-dc95-4da3-b374-f413b7186bdb\") " Nov 03 20:28:48.418985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.418666 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:48.418985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.418681 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:48.418985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.418726 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:48.418985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.418767 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:48.418985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.418951 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:48.420199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.420169 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:48.420864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.420843 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:48.421180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.421159 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:28:48.421516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.421503 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m" pod "23cbfdf8-dc95-4da3-b374-f413b7186bdb" (UID: "23cbfdf8-dc95-4da3-b374-f413b7186bdb"). InnerVolumeSpecName "kube-api-access-g567m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:48.500601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.500567 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.517226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517196 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517229 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517248 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517263 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517280 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g567m\" (UniqueName: \"kubernetes.io/projected/23cbfdf8-dc95-4da3-b374-f413b7186bdb-kube-api-access-g567m\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517294 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517306 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517314 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.517379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.517322 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cbfdf8-dc95-4da3-b374-f413b7186bdb-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.982466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:48.982430 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:49.270214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:49.270132 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25" Nov 03 20:28:49.296704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:49.296667 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:28:49.303524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:49.303500 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-9tq25"] Nov 03 20:28:49.409976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:49.409943 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" path="/var/lib/kubelet/pods/23cbfdf8-dc95-4da3-b374-f413b7186bdb/volumes" Nov 03 20:28:49.783957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:49.783919 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:50.600535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:50.600505 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:51.384651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:51.384621 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:52.053657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:52.053617 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:52.803870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:52.803831 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:53.517437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:53.517364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:53.620186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:53.620151 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:54.484200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:54.484163 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:55.274188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:55.272433 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:56.102119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:56.102083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:56.863277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:56.863246 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:57.582190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:57.582143 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:58.354991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:58.354944 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:58.909281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:58.909245 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:28:58.909530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:58.909495 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" podUID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" containerName="maas-api" containerID="cri-o://027a57666120dde7ebf1884de40921574c9647b5881561587cd3f4444b132036" gracePeriod=30 Nov 03 20:28:58.965006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:58.964969 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.260543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.259692 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.457561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.457522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.477406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.477312 5954 generic.go:356] "Generic (PLEG): container finished" podID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" containerID="027a57666120dde7ebf1884de40921574c9647b5881561587cd3f4444b132036" exitCode=0 Nov 03 20:28:59.524862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.524757 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" event={"ID":"0a32eabb-ea40-4eb5-ad0a-51c82bf34045","Type":"ContainerDied","Data":"027a57666120dde7ebf1884de40921574c9647b5881561587cd3f4444b132036"} Nov 03 20:28:59.676706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.676668 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:28:59.749570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.749535 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.839634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.827459 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\" (UniqueName: \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\") pod \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " Nov 03 20:28:59.839634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.827705 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") pod \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " Nov 03 20:28:59.839634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.827785 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\") pod \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " Nov 03 20:28:59.839634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.827816 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\") pod \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\" (UID: \"0a32eabb-ea40-4eb5-ad0a-51c82bf34045\") " Nov 03 20:28:59.840992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.840934 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca" pod "0a32eabb-ea40-4eb5-ad0a-51c82bf34045" (UID: "0a32eabb-ea40-4eb5-ad0a-51c82bf34045"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:59.842964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.842933 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls" pod "0a32eabb-ea40-4eb5-ad0a-51c82bf34045" (UID: "0a32eabb-ea40-4eb5-ad0a-51c82bf34045"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:59.853425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.851807 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q" pod "0a32eabb-ea40-4eb5-ad0a-51c82bf34045" (UID: "0a32eabb-ea40-4eb5-ad0a-51c82bf34045"). InnerVolumeSpecName "kube-api-access-s2g7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:59.856010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.855844 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs" pod "0a32eabb-ea40-4eb5-ad0a-51c82bf34045" (UID: "0a32eabb-ea40-4eb5-ad0a-51c82bf34045"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:59.930487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.930454 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g7q\" (UniqueName: \"kubernetes.io/projected/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-kube-api-access-s2g7q\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:59.930487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.930487 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:59.930487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.930498 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:59.930487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:28:59.930508 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a32eabb-ea40-4eb5-ad0a-51c82bf34045-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:00.092540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.092460 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:00.260387 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:29:00.260352 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:29:00.489874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.489843 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" Nov 03 20:29:00.490336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.489868 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg" event={"ID":"0a32eabb-ea40-4eb5-ad0a-51c82bf34045","Type":"ContainerDied","Data":"c709d08c3356dcab178831b4d981db6c6816235ec695528d9fbc946bc3c2a139"} Nov 03 20:29:00.490336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.489947 5954 scope.go:122] "RemoveContainer" containerID="027a57666120dde7ebf1884de40921574c9647b5881561587cd3f4444b132036" Nov 03 20:29:00.522835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.522802 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:29:00.526691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.526667 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d9f082-6cdb87784d-6ltwg"] Nov 03 20:29:00.584836 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:00.584782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:01.365436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:01.365382 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:01.527169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:01.527101 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" path="/var/lib/kubelet/pods/0a32eabb-ea40-4eb5-ad0a-51c82bf34045/volumes" Nov 03 20:29:02.073540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:02.073500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:02.833744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:02.833707 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:03.575214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:03.575158 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:04.292280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:04.292242 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:04.578535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:04.578439 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:04.678758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:04.678720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.179554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:05.179522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.882338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:05.882294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.327249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.318732 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.451130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.447428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.629897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.629864 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:29:06.630214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.630131 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" containerID="cri-o://9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256" gracePeriod=30 Nov 03 20:29:06.681464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.681430 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.785560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:06.785530 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:08.024748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:08.024717 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:08.566196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:08.566165 5954 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:08.566387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:08.566221 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:08.877126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:08.877083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:09.778826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:09.778789 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:10.454973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:10.454942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:10.756330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:10.756243 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:10.860545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:10.860501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:11.276866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:11.276831 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:11.938549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:11.938520 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:29:12.067484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067444 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.067672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067560 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.067672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067654 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.067826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067755 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.067826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067775 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.067826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067790 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.067992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.067864 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.068059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068021 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.068059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068032 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.068222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068167 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.068291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068246 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\") pod \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\" (UID: \"5d1cb072-9af5-4e81-9f70-c061ae37fb4c\") " Nov 03 20:29:12.068366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068346 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:12.068431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068389 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.068614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.068594 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.069240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.069215 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.069343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.069245 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.069343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.069261 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.069343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.069275 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.069343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.069290 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.070714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.070675 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:12.070973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.070930 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:12.071359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.071234 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "kube-api-access-sbzh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:12.073568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.073541 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy" pod "5d1cb072-9af5-4e81-9f70-c061ae37fb4c" (UID: "5d1cb072-9af5-4e81-9f70-c061ae37fb4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.109605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.109565 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.213134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.211537 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbzh7\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-kube-api-access-sbzh7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.213134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.211579 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.213134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.211598 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.213134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.211614 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d1cb072-9af5-4e81-9f70-c061ae37fb4c-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.639171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.639134 5954 generic.go:356] "Generic (PLEG): container finished" podID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerID="9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256" exitCode=0 Nov 03 20:29:12.639377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.639181 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" Nov 03 20:29:12.639377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.639218 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" event={"ID":"5d1cb072-9af5-4e81-9f70-c061ae37fb4c","Type":"ContainerDied","Data":"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256"} Nov 03 20:29:12.639377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.639272 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x" event={"ID":"5d1cb072-9af5-4e81-9f70-c061ae37fb4c","Type":"ContainerDied","Data":"3daa50c1605fa361574492d9a69474038578bbf324b61dabe65560084d399e39"} Nov 03 20:29:12.639377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.639294 5954 scope.go:122] "RemoveContainer" containerID="9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256" Nov 03 20:29:12.652828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.652798 5954 scope.go:122] "RemoveContainer" containerID="9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256" Nov 03 20:29:12.653154 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:29:12.653129 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256\": container with ID starting with 9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256 not found: ID does not exist" containerID="9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256" Nov 03 20:29:12.653232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.653167 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256"} err="failed to get container status \"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256\": rpc error: code = NotFound desc = could not find container \"9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256\": container with ID starting with 9f7b3dd13fc4bd84c1d12acf35ab9448858a937329815ef98d945121d4a0a256 not found: ID does not exist" Nov 03 20:29:12.667429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.667397 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:29:12.671335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.671310 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-7ws9x"] Nov 03 20:29:12.890025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.889936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.893640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:12.893616 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:29:13.425492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:13.425453 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" path="/var/lib/kubelet/pods/5d1cb072-9af5-4e81-9f70-c061ae37fb4c/volumes" Nov 03 20:29:13.968175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:13.962155 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.199303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:15.179995 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.354131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:15.352245 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:15.771132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:15.751359 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 03 20:29:15.771132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:15.751950 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 20:29:15.854129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:15.852227 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:16.007541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:16.003206 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:16.113820 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:16.113778 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:16.953935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:16.953898 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.543852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:17.536683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.650953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:17.650917 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.781705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:17.781670 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:18.699075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:18.699038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:19.798797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:19.798448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:20.589713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:20.589673 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:20.935132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:20.929183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:21.039831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:21.039784 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:21.331700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:21.331605 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:22.197271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:22.197236 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:22.983432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:22.983391 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:23.686554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:23.686503 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:24.581023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:24.580987 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:25.278873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:25.278835 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:25.973345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:25.973307 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:26.799069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:26.799030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.105306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.105275 5954 generic.go:356] "Generic (PLEG): container finished" podID="6989da16-dbc6-425e-94f5-39b8f2777431" containerID="803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7" exitCode=0 Nov 03 20:29:27.105516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.105348 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerDied","Data":"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7"} Nov 03 20:29:27.105861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.105841 5954 scope.go:122] "RemoveContainer" containerID="803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7" Nov 03 20:29:27.107797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.107766 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:27.107888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.107839 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:27.107888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.107873 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:27.345498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.345462 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:29:27.345915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.345757 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" podUID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" containerName="maas-api" containerID="cri-o://382f47927eba6c50a45f330807e0226d47a25318bc4e5a2baaa21ea7b1860bfd" gracePeriod=30 Nov 03 20:29:27.396392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.396354 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.584926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.584761 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.770181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:27.747697 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.278810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.273818 5954 generic.go:356] "Generic (PLEG): container finished" podID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" containerID="382f47927eba6c50a45f330807e0226d47a25318bc4e5a2baaa21ea7b1860bfd" exitCode=0 Nov 03 20:29:28.278810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.274017 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" event={"ID":"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7","Type":"ContainerDied","Data":"382f47927eba6c50a45f330807e0226d47a25318bc4e5a2baaa21ea7b1860bfd"} Nov 03 20:29:28.278810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.277271 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.280270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.280237 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerStarted","Data":"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d"} Nov 03 20:29:28.316596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.316549 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:29:28.373985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.372038 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:28.373985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.372096 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:28.374283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.374086 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:29:28.383074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.382986 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") pod \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " Nov 03 20:29:28.383335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.383314 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\" (UniqueName: \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\") pod \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " Nov 03 20:29:28.383555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.383540 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\") pod \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " Nov 03 20:29:28.383859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.383845 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\") pod \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\" (UID: \"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7\") " Nov 03 20:29:28.385587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.385562 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca" pod "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" (UID: "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:28.387975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.387950 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s" pod "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" (UID: "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7"). InnerVolumeSpecName "kube-api-access-8bb8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:28.389869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.389659 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls" pod "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" (UID: "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:28.395453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.394999 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs" pod "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" (UID: "6b3a20fd-5864-4a72-88ea-b7a74b6f41d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:28.398057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.398027 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.487207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.487162 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:28.487207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.487205 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:28.487207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.487217 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bb8s\" (UniqueName: \"kubernetes.io/projected/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-kube-api-access-8bb8s\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:28.487207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:28.487228 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:29.132672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.132632 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:29.291231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.291202 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" Nov 03 20:29:29.291635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.291256 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8" event={"ID":"6b3a20fd-5864-4a72-88ea-b7a74b6f41d7","Type":"ContainerDied","Data":"b8607392ad6585d8e2651796ff7476d239a922a4a82eb628f7b3058a2265a8ac"} Nov 03 20:29:29.291635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.291308 5954 scope.go:122] "RemoveContainer" containerID="382f47927eba6c50a45f330807e0226d47a25318bc4e5a2baaa21ea7b1860bfd" Nov 03 20:29:29.339064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.339027 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:29:29.345661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.345626 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7fd1a7-5d9bdcc5-4vbf8"] Nov 03 20:29:29.363919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.363884 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:29.364097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.363933 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:29.407688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.407603 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" path="/var/lib/kubelet/pods/6b3a20fd-5864-4a72-88ea-b7a74b6f41d7/volumes" Nov 03 20:29:29.832149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:29.831983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:30.266249 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:29:30.266209 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:29:30.363517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:30.363481 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:30.363931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:30.363539 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:30.489976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:30.489942 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.241130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:31.240327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.373459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:31.373424 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:31.374035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:31.374003 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:31.985042 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:31.985007 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:32.364027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:32.363986 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:32.364251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:32.364054 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:32.686431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:32.686349 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:33.364413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:33.364380 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:33.364599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:33.364429 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:33.474147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:33.463812 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.249273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:34.249242 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.368126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:34.365467 5954 patch_prober.go:29] interesting pod/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:29:34.368126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:34.365553 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Nov 03 20:29:35.088006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.087958 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.213486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.213435 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.369917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.369885 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:29:35.370396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.370231 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:29:35.370942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.370917 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:29:35.482496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.482462 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:29:35.482747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.482723 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" containerID="cri-o://c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8" gracePeriod=30 Nov 03 20:29:35.523483 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:35.523450 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:36.231249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.231212 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:29:36.232380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232359 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:36.232380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232380 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232411 5954 cpu_manager.go:414] "containerMap: removing container" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232419 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232448 5954 cpu_manager.go:414] "containerMap: removing container" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232456 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232483 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" containerName="maas-api" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232491 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" containerName="maas-api" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232505 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" containerName="maas-api" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232512 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" containerName="maas-api" Nov 03 20:29:36.232539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232540 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a8114fe2-e575-4777-89bb-9f2e1d373c68" containerName="maas-api" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232546 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8114fe2-e575-4777-89bb-9f2e1d373c68" containerName="maas-api" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232716 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232729 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cbfdf8-dc95-4da3-b374-f413b7186bdb" containerName="istio-proxy" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232739 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1cb072-9af5-4e81-9f70-c061ae37fb4c" containerName="istio-proxy" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232759 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a32eabb-ea40-4eb5-ad0a-51c82bf34045" containerName="maas-api" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232766 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3a20fd-5864-4a72-88ea-b7a74b6f41d7" containerName="maas-api" Nov 03 20:29:36.232838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.232774 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8114fe2-e575-4777-89bb-9f2e1d373c68" containerName="maas-api" Nov 03 20:29:36.235531 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.235513 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.247990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.247964 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:29:36.268476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.268446 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a744eb-openshift-default-dockercfg-n26cv\"" Nov 03 20:29:36.343168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343080 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.343368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343302 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.343368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343333 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.343560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343524 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.343740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343675 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.343915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343886 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pf2g\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.344033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.343970 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.344033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.344018 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.344491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.344078 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.395795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.395761 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:36.462133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.446403 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pf2g\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.475438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.475295 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.475876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.475846 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.476282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.476258 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.476608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.476588 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.476876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.476859 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.477014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.476995 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.477278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.477259 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.477454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.477437 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.480569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.480543 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.488413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.480914 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.489301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.481466 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.489509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.482094 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.489646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.482584 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.509446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.509370 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.535747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.535706 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pf2g\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.537592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.537531 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.538278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.538216 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.551862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.551819 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:36.850617 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:29:36.850584 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf20803_6403_415c_b166_27058ff3e898.slice/crio-d747e012ec159fe210c105a463f531a0e62bc9eb34722b6ae7031ed7358f75f6 WatchSource:0}: Error finding container d747e012ec159fe210c105a463f531a0e62bc9eb34722b6ae7031ed7358f75f6: Status 404 returned error can't find the container with id d747e012ec159fe210c105a463f531a0e62bc9eb34722b6ae7031ed7358f75f6 Nov 03 20:29:36.850740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.850684 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:29:36.853756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.853724 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:36.853845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.853788 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:36.853845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:36.853818 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:37.107631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.107594 5954 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:37.107819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.107656 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:37.165377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.165341 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:37.522194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.522098 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" event={"ID":"eaf20803-6403-415c-b166-27058ff3e898","Type":"ContainerStarted","Data":"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14"} Nov 03 20:29:37.522194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.522172 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" event={"ID":"eaf20803-6403-415c-b166-27058ff3e898","Type":"ContainerStarted","Data":"d747e012ec159fe210c105a463f531a0e62bc9eb34722b6ae7031ed7358f75f6"} Nov 03 20:29:37.549579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.549529 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podStartSLOduration=1.549515418 podStartE2EDuration="1.549515418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:37.544377204 +0000 UTC m=+1918.269384801" watchObservedRunningTime="2026-11-03 20:29:37.549515418 +0000 UTC m=+1918.274523011" Nov 03 20:29:37.552459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.552434 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:37.553911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.553888 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:37.554010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:37.553934 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:38.090196 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:38.090161 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:38.553273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:38.553235 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:38.553683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:38.553295 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:38.902958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:38.902920 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.047126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:39.046812 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.236188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:39.235536 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.552822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:39.552747 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:39.552822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:39.552804 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:39.805774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:39.805685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:40.494229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:40.494197 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:40.562076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:40.558546 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:40.562076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:40.558615 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:40.966469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:40.966443 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:29:41.047511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047412 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.047654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047610 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.047707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047691 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.047854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047835 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.047903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047879 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.047953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047937 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.048004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.047989 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.048156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048124 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:41.048394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048268 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:41.048529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048391 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:41.048653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048412 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:41.048778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048165 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.049017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.048997 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\") pod \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\" (UID: \"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8\") " Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.049420 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050203 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050226 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050241 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050255 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.050563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050271 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.050944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.050873 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "kube-api-access-nrnrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:41.051505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.051480 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:41.052175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.052154 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:41.056629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.056607 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token" pod "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" (UID: "4338e6dd-f6fa-45a2-9e84-0a87e6d262a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:41.151827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.151795 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.151827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.151823 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.151827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.151834 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.152087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.151843 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrnrk\" (UniqueName: \"kubernetes.io/projected/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8-kube-api-access-nrnrk\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:41.194691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.194649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:41.552656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.552610 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:41.552877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.552677 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:41.687504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.687408 5954 generic.go:356] "Generic (PLEG): container finished" podID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerID="c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8" exitCode=0 Nov 03 20:29:41.687504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.687491 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" Nov 03 20:29:41.687743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.687492 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerDied","Data":"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8"} Nov 03 20:29:41.687743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.687657 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm" event={"ID":"4338e6dd-f6fa-45a2-9e84-0a87e6d262a8","Type":"ContainerDied","Data":"68820c43ba536802ef65069bd34363312483ab6584d18e43c6bbcf143e3428ff"} Nov 03 20:29:41.687743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.687682 5954 scope.go:122] "RemoveContainer" containerID="c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8" Nov 03 20:29:41.699512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.699482 5954 scope.go:122] "RemoveContainer" containerID="f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9" Nov 03 20:29:41.711166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.711139 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.724489 5954 scope.go:122] "RemoveContainer" containerID="c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8" Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:29:41.724940 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8\": container with ID starting with c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8 not found: ID does not exist" containerID="c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8" Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.724980 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8"} err="failed to get container status \"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8\": rpc error: code = NotFound desc = could not find container \"c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8\": container with ID starting with c2a252086feb05fcd2af3c7ab4f6b65778a7811e7fc9059d1be8b1101bf4b9b8 not found: ID does not exist" Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.725006 5954 scope.go:122] "RemoveContainer" containerID="f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9" Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:29:41.725309 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9\": container with ID starting with f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9 not found: ID does not exist" containerID="f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9" Nov 03 20:29:41.728155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.725336 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9"} err="failed to get container status \"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9\": rpc error: code = NotFound desc = could not find container \"f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9\": container with ID starting with f52e34ce346d93b40c06e41aba1faf1aff34eaa97bf8eab2fe132ec437fb08c9 not found: ID does not exist" Nov 03 20:29:41.738179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.737027 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7fd1a7-openshift-default-76bfc9f56f-kpzfm"] Nov 03 20:29:41.989184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:41.989080 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:42.552605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:42.552572 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:42.552798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:42.552658 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:42.738135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:42.735685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:43.409272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:43.409241 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" path="/var/lib/kubelet/pods/4338e6dd-f6fa-45a2-9e84-0a87e6d262a8/volumes" Nov 03 20:29:43.436301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:43.436261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:43.563322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:43.563280 5954 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:29:43.564223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:43.564001 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 20:29:44.174686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:44.174649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:44.558168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:44.558059 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:44.558571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:44.558552 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:44.559462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:44.559443 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:29:44.980954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:44.980917 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.107136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:45.101686 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.695638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:45.695570 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.938813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:45.938776 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.212063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:46.212021 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.311918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:46.311881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.483185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:46.481636 5954 scope.go:122] "RemoveContainer" containerID="6dcc59cefe0ddaea4ffcae8f6a4fc112808177c3baa026db90918a266b8f9869" Nov 03 20:29:46.604131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:46.598676 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:47.448071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:47.448041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:48.123552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:48.123514 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:48.568520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:48.568428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:48.814426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:48.814394 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:49.497935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:49.497900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:50.191936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:50.191888 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:51.040275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:51.040241 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:52.093865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:52.093820 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:52.891457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:52.891420 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:53.738339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.738305 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:29:53.740101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740066 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:29:53.740101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740100 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:29:53.740299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740190 5954 cpu_manager.go:414] "containerMap: removing container" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:53.740299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740198 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:53.740432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740417 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:29:53.740479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740435 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd43e8ac-3fd7-49df-b061-f55be5c48375" containerName="istio-proxy" Nov 03 20:29:53.740479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.740453 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:29:53.746340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.746318 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.756395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.756365 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:29:53.777970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.777929 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:53.804560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.804515 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.804819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.804576 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.804819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.804796 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.804983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.804862 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.804983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.804895 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.805092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.805029 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.805199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.805177 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9tzg\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.805284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.805263 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.805336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.805317 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.911618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.911572 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9tzg\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.911808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.911668 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.911808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.911730 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.911965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.911937 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.912201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.912017 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.912201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.912158 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.912354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.912202 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.912354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.912233 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.912354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.912317 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.913320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.913290 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.914584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.914484 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.914740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.914715 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.914942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.914830 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.915486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.915449 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.917384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.917361 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.917541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.917525 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.925933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.925899 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:53.926218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:53.926194 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9tzg\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:54.067085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.066982 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:54.368153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.368123 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:29:54.370236 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:29:54.370204 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175752d9_a28f_4352_9e81_5dc71de5df45.slice/crio-f51106f8aff0f9803901342de6aa091434af35bae4888143a86e81b2055df4e1 WatchSource:0}: Error finding container f51106f8aff0f9803901342de6aa091434af35bae4888143a86e81b2055df4e1: Status 404 returned error can't find the container with id f51106f8aff0f9803901342de6aa091434af35bae4888143a86e81b2055df4e1 Nov 03 20:29:54.373748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.373715 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:54.373848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.373791 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:54.373848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.373822 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:29:54.583549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:54.583500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:55.079261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:55.079206 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" event={"ID":"175752d9-a28f-4352-9e81-5dc71de5df45","Type":"ContainerStarted","Data":"8d43f9f53a7d3cfecf0b11e8ced37b98e5aa577e9aa282078d1743c6a560e7bf"} Nov 03 20:29:55.079798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:55.079282 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" event={"ID":"175752d9-a28f-4352-9e81-5dc71de5df45","Type":"ContainerStarted","Data":"f51106f8aff0f9803901342de6aa091434af35bae4888143a86e81b2055df4e1"} Nov 03 20:29:55.105884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:55.105829 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podStartSLOduration=2.105814294 podStartE2EDuration="2.105814294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:55.102354607 +0000 UTC m=+1935.827362216" watchObservedRunningTime="2026-11-03 20:29:55.105814294 +0000 UTC m=+1935.830821949" Nov 03 20:29:55.275085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:55.275052 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:56.067321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:56.067278 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:29:56.069266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:56.069237 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:29:56.069416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:56.069291 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:29:56.087858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:56.087814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:56.886825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:56.886790 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:57.080760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:57.080532 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:29:57.080760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:57.080599 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:29:57.683135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:57.674779 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:58.067700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:58.067624 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:29:58.067700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:58.067672 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:29:58.309027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:58.308994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.039662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.039619 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.067835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.067799 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:29:59.068003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.067850 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:29:59.559660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.559618 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:29:59.559928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.559875 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" podUID="de8d6ef3-890b-4146-95a1-f9367f7194b0" containerName="maas-api" containerID="cri-o://c14f9ae5b2d0789899f9bf539417285c2774d0855dcb402868d029f2a24b4549" gracePeriod=30 Nov 03 20:29:59.640250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.640205 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.804126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:29:59.800960 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.189129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.185248 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:30:00.189129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.185390 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:30:00.381376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.333468 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.439067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.438563 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:00.479866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.479793 5954 generic.go:356] "Generic (PLEG): container finished" podID="de8d6ef3-890b-4146-95a1-f9367f7194b0" containerID="c14f9ae5b2d0789899f9bf539417285c2774d0855dcb402868d029f2a24b4549" exitCode=0 Nov 03 20:30:00.494137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.488810 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:30:00.494137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.488838 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4338e6dd-f6fa-45a2-9e84-0a87e6d262a8" containerName="istio-proxy" Nov 03 20:30:00.505819 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:00.505784 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:30:00.553390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.553179 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" event={"ID":"de8d6ef3-890b-4146-95a1-f9367f7194b0","Type":"ContainerDied","Data":"c14f9ae5b2d0789899f9bf539417285c2774d0855dcb402868d029f2a24b4549"} Nov 03 20:30:00.553390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.553238 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:00.553673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.553610 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.557914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.557892 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:30:00.560637 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.560614 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:00.560765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.560657 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:00.560826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.560777 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.565766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.565748 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:00.565975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.565959 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.571211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.571191 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:00.571335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.571228 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:00.571590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.571472 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.576201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.576180 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:00.576316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.576262 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.580600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.580582 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:00.580731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.580611 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:00.580731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.580629 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:00.580860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.580729 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.585741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.585722 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:00.585865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.585760 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:00.585865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.585794 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:00.585865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.585822 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:00.585865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.585848 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.611972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.611784 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:30:00.620178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.612408 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.620178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.618340 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:00.620178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.618460 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.702140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.701701 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\") pod \"de8d6ef3-890b-4146-95a1-f9367f7194b0\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " Nov 03 20:30:00.702140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.701943 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\") pod \"de8d6ef3-890b-4146-95a1-f9367f7194b0\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " Nov 03 20:30:00.702140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702012 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\" (UniqueName: \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\") pod \"de8d6ef3-890b-4146-95a1-f9367f7194b0\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702579 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702646 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqhd\" (UniqueName: \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702673 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702707 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702732 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702763 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrts\" (UniqueName: \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.702912 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcfx\" (UniqueName: \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703124 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4vf\" (UniqueName: \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703198 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzmh\" (UniqueName: \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703273 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703336 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703367 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703476 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrqz\" (UniqueName: \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703538 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2k75\" (UniqueName: \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703617 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.703935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.703641 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sb7\" (UniqueName: \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.704962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.704941 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca" pod "de8d6ef3-890b-4146-95a1-f9367f7194b0" (UID: "de8d6ef3-890b-4146-95a1-f9367f7194b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:00.740429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.740381 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs" pod "de8d6ef3-890b-4146-95a1-f9367f7194b0" (UID: "de8d6ef3-890b-4146-95a1-f9367f7194b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:00.768594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.768560 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm" pod "de8d6ef3-890b-4146-95a1-f9367f7194b0" (UID: "de8d6ef3-890b-4146-95a1-f9367f7194b0"). InnerVolumeSpecName "kube-api-access-rnhdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:00.773630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.773061 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.806515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.806214 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\") pod \"de8d6ef3-890b-4146-95a1-f9367f7194b0\" (UID: \"de8d6ef3-890b-4146-95a1-f9367f7194b0\") " Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.806989 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807042 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807063 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807148 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrqz\" (UniqueName: \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807186 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2k75\" (UniqueName: \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807232 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807249 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sb7\" (UniqueName: \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807370 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807397 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqhd\" (UniqueName: \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807414 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807439 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807460 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807480 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrts\" (UniqueName: \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807515 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcfx\" (UniqueName: \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807631 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4vf\" (UniqueName: \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807670 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzmh\" (UniqueName: \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807770 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807787 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de8d6ef3-890b-4146-95a1-f9367f7194b0-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:00.808352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.807796 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnhdm\" (UniqueName: \"kubernetes.io/projected/de8d6ef3-890b-4146-95a1-f9367f7194b0-kube-api-access-rnhdm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:00.810331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.809232 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.810331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.810211 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.811192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.810960 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.811192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.811149 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.811532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.811320 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.811744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.811723 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.811939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.811906 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.812018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.812002 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls" pod "de8d6ef3-890b-4146-95a1-f9367f7194b0" (UID: "de8d6ef3-890b-4146-95a1-f9367f7194b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:00.812610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.812594 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.851321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.848631 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqhd\" (UniqueName: \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.851321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.848689 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrts\" (UniqueName: \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\") pod \"maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:00.854155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.853807 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcfx\" (UniqueName: \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\") pod \"maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:00.854321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.854276 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sb7\" (UniqueName: \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:00.854767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.854744 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2k75\" (UniqueName: \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\") pod \"maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:00.854898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.854874 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzmh\" (UniqueName: \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\") pod \"maas-api-key-cleanup-29895630-8m8lw\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.855270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.855252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrqz\" (UniqueName: \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\") pod \"maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:00.855544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.855517 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:00.855856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.855830 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4vf\" (UniqueName: \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:00.911627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.910714 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de8d6ef3-890b-4146-95a1-f9367f7194b0-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:00.921695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.921663 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:00.965397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:00.964171 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:01.009149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.004394 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:01.032993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.032676 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:01.077709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.077027 5954 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 20:30:01.077709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.077090 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 20:30:01.078301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.078272 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:01.099889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.097703 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:01.124350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.124134 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:01.211905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.211861 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:01.225358 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:30:01.225311 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a64efc_6e67_4742_8542_b6b49ea06f98.slice/crio-8aa8bd783f0e54b9ad392a78f05816bf08c5d4d44b1194682f823d3f328e935a WatchSource:0}: Error finding container 8aa8bd783f0e54b9ad392a78f05816bf08c5d4d44b1194682f823d3f328e935a: Status 404 returned error can't find the container with id 8aa8bd783f0e54b9ad392a78f05816bf08c5d4d44b1194682f823d3f328e935a Nov 03 20:30:01.309851 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:30:01.309819 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff04b177_e9bb_497e_b644_48e0ac2b758b.slice/crio-66a57de0ce28a34978144a0a02ebba28813aadf1e29342c04bc2c3a4ec61151c WatchSource:0}: Error finding container 66a57de0ce28a34978144a0a02ebba28813aadf1e29342c04bc2c3a4ec61151c: Status 404 returned error can't find the container with id 66a57de0ce28a34978144a0a02ebba28813aadf1e29342c04bc2c3a4ec61151c Nov 03 20:30:01.310160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.310137 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:01.418366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.418339 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:01.418577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.418559 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:01.419215 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:30:01.419183 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f42e1a_5464_4d1f_be9d_6e50c12c9c1e.slice/crio-62cfcec19f2815df5cb388ef0f719717a93bbb5c6b9d5c32df5f2088922add7a WatchSource:0}: Error finding container 62cfcec19f2815df5cb388ef0f719717a93bbb5c6b9d5c32df5f2088922add7a: Status 404 returned error can't find the container with id 62cfcec19f2815df5cb388ef0f719717a93bbb5c6b9d5c32df5f2088922add7a Nov 03 20:30:01.464584 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:30:01.464525 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f838a86_cd37_469a_90e7_47480ce31aff.slice/crio-91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20 WatchSource:0}: Error finding container 91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20: Status 404 returned error can't find the container with id 91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20 Nov 03 20:30:01.470745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.469073 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:01.497541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.497452 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:01.507058 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.507026 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:01.525932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.525900 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:01.533066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.533028 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerStarted","Data":"66a57de0ce28a34978144a0a02ebba28813aadf1e29342c04bc2c3a4ec61151c"} Nov 03 20:30:01.534507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.534483 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerStarted","Data":"91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20"} Nov 03 20:30:01.535789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.535765 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerStarted","Data":"62cfcec19f2815df5cb388ef0f719717a93bbb5c6b9d5c32df5f2088922add7a"} Nov 03 20:30:01.536992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.536963 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerStarted","Data":"8aa8bd783f0e54b9ad392a78f05816bf08c5d4d44b1194682f823d3f328e935a"} Nov 03 20:30:01.537958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.537929 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerStarted","Data":"1ca4d8c15d39795fff9cfc797f622c1907314c81c6b7e118b5f169c1dbfe44b6"} Nov 03 20:30:01.538934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.538911 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" event={"ID":"de8d6ef3-890b-4146-95a1-f9367f7194b0","Type":"ContainerDied","Data":"5e36f43f5d940d35e2bbdd5d9b4c921de84c53ad5ce39bd69cba04a62cefd2af"} Nov 03 20:30:01.539035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.538951 5954 scope.go:122] "RemoveContainer" containerID="c14f9ae5b2d0789899f9bf539417285c2774d0855dcb402868d029f2a24b4549" Nov 03 20:30:01.539035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.538979 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr" Nov 03 20:30:01.568406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.568361 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:30:01.572313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.572286 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-683ba6-8466f44768-xgxkr"] Nov 03 20:30:01.620195 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:30:01.620156 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5a9d56_ffd2_4307_832a_30afacf4b48a.slice/crio-30e01d4aebcb451b395df30edc5660d3d4709f90c282f5614d4acce02da0a2ed WatchSource:0}: Error finding container 30e01d4aebcb451b395df30edc5660d3d4709f90c282f5614d4acce02da0a2ed: Status 404 returned error can't find the container with id 30e01d4aebcb451b395df30edc5660d3d4709f90c282f5614d4acce02da0a2ed Nov 03 20:30:01.644445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:01.644409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:02.073359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.073331 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:30:02.073766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.073732 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:30:02.074517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.074489 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:30:02.481243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.481207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:02.609005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.608903 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerStarted","Data":"d16338c4cdd1f0b6a3e72daa8840e152d6b8e3a6e23f901fedd35aff6d2f9a0c"} Nov 03 20:30:02.611503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.611423 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerStarted","Data":"2bce34eee7c6fbe325a5061b6bb07888df90b486975e806975f75723b1051114"} Nov 03 20:30:02.615641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.614378 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerStarted","Data":"2bd799e8adc477c84ee23eabf13ab9c9cd8f9bb11918c80e2f5eae63c9334aa3"} Nov 03 20:30:02.618025 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.617997 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerStarted","Data":"b793fee7a3f7e817d1cf51e521e75c6b33e9c8a42323cef398db4b56c436604b"} Nov 03 20:30:02.625771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.625736 5954 generic.go:356] "Generic (PLEG): container finished" podID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerID="34169d634cd13d16bdc49d4f74dcd359dc251dc4b6258dc94b9e19d96b3534b3" exitCode=60 Nov 03 20:30:02.626062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.626039 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerDied","Data":"34169d634cd13d16bdc49d4f74dcd359dc251dc4b6258dc94b9e19d96b3534b3"} Nov 03 20:30:02.626745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.626725 5954 scope.go:122] "RemoveContainer" containerID="34169d634cd13d16bdc49d4f74dcd359dc251dc4b6258dc94b9e19d96b3534b3" Nov 03 20:30:02.639122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.639053 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" podStartSLOduration=2.639034934 podStartE2EDuration="2.639034934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.633345704 +0000 UTC m=+1943.358353301" watchObservedRunningTime="2026-11-03 20:30:02.639034934 +0000 UTC m=+1943.364042532" Nov 03 20:30:02.642713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.640014 5954 generic.go:356] "Generic (PLEG): container finished" podID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerID="1c4ef97d182bd5188dd61ce9cf36e2d9a8e813a0c54902cac2e6545c59b50b1c" exitCode=60 Nov 03 20:30:02.642713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.640603 5954 scope.go:122] "RemoveContainer" containerID="1c4ef97d182bd5188dd61ce9cf36e2d9a8e813a0c54902cac2e6545c59b50b1c" Nov 03 20:30:02.642713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.641144 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerDied","Data":"1c4ef97d182bd5188dd61ce9cf36e2d9a8e813a0c54902cac2e6545c59b50b1c"} Nov 03 20:30:02.646644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.646618 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerStarted","Data":"30e01d4aebcb451b395df30edc5660d3d4709f90c282f5614d4acce02da0a2ed"} Nov 03 20:30:02.650898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.650174 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerStarted","Data":"4942e20e169a2ad4371c80764f43e94f3dfed467709d3d083ddec466941ed7ea"} Nov 03 20:30:02.677690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.677644 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" podStartSLOduration=2.677629392 podStartE2EDuration="2.677629392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.673365506 +0000 UTC m=+1943.398373103" watchObservedRunningTime="2026-11-03 20:30:02.677629392 +0000 UTC m=+1943.402636989" Nov 03 20:30:02.696811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:02.696765 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" podStartSLOduration=2.696751338 podStartE2EDuration="2.696751338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.690933062 +0000 UTC m=+1943.415940659" watchObservedRunningTime="2026-11-03 20:30:02.696751338 +0000 UTC m=+1943.421758930" Nov 03 20:30:03.397307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.397273 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:03.408628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.408600 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8d6ef3-890b-4146-95a1-f9367f7194b0" path="/var/lib/kubelet/pods/de8d6ef3-890b-4146-95a1-f9367f7194b0/volumes" Nov 03 20:30:03.661430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.661399 5954 generic.go:356] "Generic (PLEG): container finished" podID="2f838a86-cd37-469a-90e7-47480ce31aff" containerID="b793fee7a3f7e817d1cf51e521e75c6b33e9c8a42323cef398db4b56c436604b" exitCode=60 Nov 03 20:30:03.661873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.661489 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerDied","Data":"b793fee7a3f7e817d1cf51e521e75c6b33e9c8a42323cef398db4b56c436604b"} Nov 03 20:30:03.661873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.661834 5954 scope.go:122] "RemoveContainer" containerID="b793fee7a3f7e817d1cf51e521e75c6b33e9c8a42323cef398db4b56c436604b" Nov 03 20:30:03.663828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.663795 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerStarted","Data":"00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e"} Nov 03 20:30:03.666043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.665942 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerStarted","Data":"f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e"} Nov 03 20:30:03.667770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.667744 5954 generic.go:356] "Generic (PLEG): container finished" podID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerID="eea6068017f7842f81b56badc7d87106ec7586b29f639879eaa199dddcf97add" exitCode=60 Nov 03 20:30:03.667878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.667846 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerDied","Data":"eea6068017f7842f81b56badc7d87106ec7586b29f639879eaa199dddcf97add"} Nov 03 20:30:03.668148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.668096 5954 scope.go:122] "RemoveContainer" containerID="eea6068017f7842f81b56badc7d87106ec7586b29f639879eaa199dddcf97add" Nov 03 20:30:03.669542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.669511 5954 generic.go:356] "Generic (PLEG): container finished" podID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerID="4942e20e169a2ad4371c80764f43e94f3dfed467709d3d083ddec466941ed7ea" exitCode=60 Nov 03 20:30:03.669717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.669696 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerDied","Data":"4942e20e169a2ad4371c80764f43e94f3dfed467709d3d083ddec466941ed7ea"} Nov 03 20:30:03.669868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.669839 5954 scope.go:122] "RemoveContainer" containerID="4942e20e169a2ad4371c80764f43e94f3dfed467709d3d083ddec466941ed7ea" Nov 03 20:30:03.671943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.671921 5954 generic.go:356] "Generic (PLEG): container finished" podID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerID="e626c4989754df1fb63f49892036b29b20be2507d220d1dce6cf12f3f23c3ed0" exitCode=60 Nov 03 20:30:03.672048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.671999 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerDied","Data":"e626c4989754df1fb63f49892036b29b20be2507d220d1dce6cf12f3f23c3ed0"} Nov 03 20:30:03.672201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.672187 5954 scope.go:122] "RemoveContainer" containerID="e626c4989754df1fb63f49892036b29b20be2507d220d1dce6cf12f3f23c3ed0" Nov 03 20:30:03.673786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.673767 5954 generic.go:356] "Generic (PLEG): container finished" podID="06832e97-39ca-478b-91af-70eacec17a09" containerID="029e2298e3dcd40ed2a9ef6d0151c9e7ae76280822c28d3d41afcea0ed7b2aa1" exitCode=60 Nov 03 20:30:03.673900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.673854 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerDied","Data":"029e2298e3dcd40ed2a9ef6d0151c9e7ae76280822c28d3d41afcea0ed7b2aa1"} Nov 03 20:30:03.674002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.673986 5954 scope.go:122] "RemoveContainer" containerID="029e2298e3dcd40ed2a9ef6d0151c9e7ae76280822c28d3d41afcea0ed7b2aa1" Nov 03 20:30:03.679342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.679307 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerID="2bd799e8adc477c84ee23eabf13ab9c9cd8f9bb11918c80e2f5eae63c9334aa3" exitCode=60 Nov 03 20:30:03.682121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.679945 5954 scope.go:122] "RemoveContainer" containerID="2bd799e8adc477c84ee23eabf13ab9c9cd8f9bb11918c80e2f5eae63c9334aa3" Nov 03 20:30:03.682121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.679942 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerDied","Data":"2bd799e8adc477c84ee23eabf13ab9c9cd8f9bb11918c80e2f5eae63c9334aa3"} Nov 03 20:30:03.726710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.726638 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" podStartSLOduration=3.726619662 podStartE2EDuration="3.726619662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.725302948 +0000 UTC m=+1944.450310545" watchObservedRunningTime="2026-11-03 20:30:03.726619662 +0000 UTC m=+1944.451627258" Nov 03 20:30:03.784910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:03.784849 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" podStartSLOduration=3.784831954 podStartE2EDuration="3.784831954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.782244936 +0000 UTC m=+1944.507252536" watchObservedRunningTime="2026-11-03 20:30:03.784831954 +0000 UTC m=+1944.509839552" Nov 03 20:30:04.165123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.165066 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:04.690351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.690308 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerStarted","Data":"b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef"} Nov 03 20:30:04.692515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.692488 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerStarted","Data":"408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de"} Nov 03 20:30:04.694416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.694389 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerStarted","Data":"1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62"} Nov 03 20:30:04.696323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.696300 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerStarted","Data":"aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64"} Nov 03 20:30:04.698478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.698454 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerStarted","Data":"01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe"} Nov 03 20:30:04.700286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.700264 5954 generic.go:356] "Generic (PLEG): container finished" podID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerID="00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e" exitCode=60 Nov 03 20:30:04.700389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.700342 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerDied","Data":"00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e"} Nov 03 20:30:04.700440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.700394 5954 scope.go:122] "RemoveContainer" containerID="34169d634cd13d16bdc49d4f74dcd359dc251dc4b6258dc94b9e19d96b3534b3" Nov 03 20:30:04.700525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.700506 5954 scope.go:122] "RemoveContainer" containerID="00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e" Nov 03 20:30:04.700772 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:04.700753 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2_odh-ai-gateway-infra(62f42e1a-5464-4d1f-be9d-6e50c12c9c1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" Nov 03 20:30:04.702310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.702289 5954 generic.go:356] "Generic (PLEG): container finished" podID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerID="f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e" exitCode=60 Nov 03 20:30:04.702395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.702352 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerDied","Data":"f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e"} Nov 03 20:30:04.702571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.702553 5954 scope.go:122] "RemoveContainer" containerID="f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e" Nov 03 20:30:04.702837 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:04.702815 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q_odh-ai-gateway-infra(17a64efc-6e67-4742-8542-b6b49ea06f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" Nov 03 20:30:04.704253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.704232 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerStarted","Data":"6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5"} Nov 03 20:30:04.719273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.719246 5954 scope.go:122] "RemoveContainer" containerID="1c4ef97d182bd5188dd61ce9cf36e2d9a8e813a0c54902cac2e6545c59b50b1c" Nov 03 20:30:04.739294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.739241 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" podStartSLOduration=4.739223266 podStartE2EDuration="4.739223266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.7341808 +0000 UTC m=+1945.459188400" watchObservedRunningTime="2026-11-03 20:30:04.739223266 +0000 UTC m=+1945.464230866" Nov 03 20:30:04.758509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.758466 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" podStartSLOduration=4.75845346 podStartE2EDuration="4.75845346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.754188812 +0000 UTC m=+1945.479196410" watchObservedRunningTime="2026-11-03 20:30:04.75845346 +0000 UTC m=+1945.483461095" Nov 03 20:30:04.812787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:04.812744 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" podStartSLOduration=4.81273071 podStartE2EDuration="4.81273071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.809747253 +0000 UTC m=+1945.534754850" watchObservedRunningTime="2026-11-03 20:30:04.81273071 +0000 UTC m=+1945.537738306" Nov 03 20:30:05.097164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.097085 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:05.714648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.714611 5954 generic.go:356] "Generic (PLEG): container finished" podID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerID="408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de" exitCode=60 Nov 03 20:30:05.715099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.714687 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerDied","Data":"408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de"} Nov 03 20:30:05.715099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.714751 5954 scope.go:122] "RemoveContainer" containerID="e626c4989754df1fb63f49892036b29b20be2507d220d1dce6cf12f3f23c3ed0" Nov 03 20:30:05.715099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.715062 5954 scope.go:122] "RemoveContainer" containerID="408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de" Nov 03 20:30:05.715543 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.715518 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh_odh-ai-gateway-infra(7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" Nov 03 20:30:05.716947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.716925 5954 generic.go:356] "Generic (PLEG): container finished" podID="06832e97-39ca-478b-91af-70eacec17a09" containerID="1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62" exitCode=60 Nov 03 20:30:05.717019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.716987 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerDied","Data":"1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62"} Nov 03 20:30:05.717315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.717301 5954 scope.go:122] "RemoveContainer" containerID="1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62" Nov 03 20:30:05.717713 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.717690 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2_odh-ai-gateway-infra(06832e97-39ca-478b-91af-70eacec17a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" podUID="06832e97-39ca-478b-91af-70eacec17a09" Nov 03 20:30:05.719140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.719082 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerID="aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64" exitCode=60 Nov 03 20:30:05.719224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.719134 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerDied","Data":"aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64"} Nov 03 20:30:05.719439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.719425 5954 scope.go:122] "RemoveContainer" containerID="aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64" Nov 03 20:30:05.719786 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.719768 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9_odh-ai-gateway-infra(ff04b177-e9bb-497e-b644-48e0ac2b758b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" Nov 03 20:30:05.721199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.721171 5954 generic.go:356] "Generic (PLEG): container finished" podID="2f838a86-cd37-469a-90e7-47480ce31aff" containerID="01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe" exitCode=60 Nov 03 20:30:05.721286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.721223 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerDied","Data":"01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe"} Nov 03 20:30:05.721587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.721564 5954 scope.go:122] "RemoveContainer" containerID="01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe" Nov 03 20:30:05.722051 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.722025 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq_odh-ai-gateway-infra(2f838a86-cd37-469a-90e7-47480ce31aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" Nov 03 20:30:05.723197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.723177 5954 scope.go:122] "RemoveContainer" containerID="00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e" Nov 03 20:30:05.723562 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.723526 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2_odh-ai-gateway-infra(62f42e1a-5464-4d1f-be9d-6e50c12c9c1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" Nov 03 20:30:05.724892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.724875 5954 scope.go:122] "RemoveContainer" containerID="f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e" Nov 03 20:30:05.725278 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.725252 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q_odh-ai-gateway-infra(17a64efc-6e67-4742-8542-b6b49ea06f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" Nov 03 20:30:05.727272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.727248 5954 generic.go:356] "Generic (PLEG): container finished" podID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerID="6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5" exitCode=60 Nov 03 20:30:05.727395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.727379 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerDied","Data":"6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5"} Nov 03 20:30:05.727641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.727624 5954 scope.go:122] "RemoveContainer" containerID="6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5" Nov 03 20:30:05.728304 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.728276 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-8m8lw_odh-ai-gateway-infra(8a5a9d56-ffd2-4307-832a-30afacf4b48a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" Nov 03 20:30:05.729694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.729673 5954 generic.go:356] "Generic (PLEG): container finished" podID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerID="b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef" exitCode=60 Nov 03 20:30:05.729795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.729760 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerDied","Data":"b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef"} Nov 03 20:30:05.730071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.730056 5954 scope.go:122] "RemoveContainer" containerID="b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef" Nov 03 20:30:05.730386 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:05.730364 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx_odh-ai-gateway-infra(f04aa9d5-906e-4480-a073-6c28434e1a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" Nov 03 20:30:05.731620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.731600 5954 scope.go:122] "RemoveContainer" containerID="029e2298e3dcd40ed2a9ef6d0151c9e7ae76280822c28d3d41afcea0ed7b2aa1" Nov 03 20:30:05.747765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.747743 5954 scope.go:122] "RemoveContainer" containerID="2bd799e8adc477c84ee23eabf13ab9c9cd8f9bb11918c80e2f5eae63c9334aa3" Nov 03 20:30:05.764363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.764347 5954 scope.go:122] "RemoveContainer" containerID="b793fee7a3f7e817d1cf51e521e75c6b33e9c8a42323cef398db4b56c436604b" Nov 03 20:30:05.778855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.778834 5954 scope.go:122] "RemoveContainer" containerID="eea6068017f7842f81b56badc7d87106ec7586b29f639879eaa199dddcf97add" Nov 03 20:30:05.792799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:05.792779 5954 scope.go:122] "RemoveContainer" containerID="4942e20e169a2ad4371c80764f43e94f3dfed467709d3d083ddec466941ed7ea" Nov 03 20:30:06.022057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.021977 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.743897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.743872 5954 scope.go:122] "RemoveContainer" containerID="b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef" Nov 03 20:30:06.744808 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.744541 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx_odh-ai-gateway-infra(f04aa9d5-906e-4480-a073-6c28434e1a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" Nov 03 20:30:06.746500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.746474 5954 scope.go:122] "RemoveContainer" containerID="408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de" Nov 03 20:30:06.746961 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.746922 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh_odh-ai-gateway-infra(7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" Nov 03 20:30:06.748276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.748255 5954 scope.go:122] "RemoveContainer" containerID="1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62" Nov 03 20:30:06.748611 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.748589 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2_odh-ai-gateway-infra(06832e97-39ca-478b-91af-70eacec17a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" podUID="06832e97-39ca-478b-91af-70eacec17a09" Nov 03 20:30:06.749884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.749863 5954 scope.go:122] "RemoveContainer" containerID="aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64" Nov 03 20:30:06.750243 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.750209 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9_odh-ai-gateway-infra(ff04b177-e9bb-497e-b644-48e0ac2b758b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" Nov 03 20:30:06.751529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.751508 5954 scope.go:122] "RemoveContainer" containerID="01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe" Nov 03 20:30:06.751867 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.751845 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq_odh-ai-gateway-infra(2f838a86-cd37-469a-90e7-47480ce31aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" Nov 03 20:30:06.753052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.753033 5954 scope.go:122] "RemoveContainer" containerID="6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5" Nov 03 20:30:06.753341 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:06.753320 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-8m8lw_odh-ai-gateway-infra(8a5a9d56-ffd2-4307-832a-30afacf4b48a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" Nov 03 20:30:06.787174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:06.787148 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:07.506123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:07.506075 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:07.610560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:07.610528 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:07.836811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:07.836730 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:30:07.837328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:07.836975 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" containerID="cri-o://6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729" gracePeriod=30 Nov 03 20:30:07.888597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:07.888568 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:08.379140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:08.377879 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:08.850047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:08.849910 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:09.870312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:09.870277 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:10.092262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:10.092221 5954 patch_prober.go:29] interesting pod/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:30:10.092440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:10.092284 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:30:10.575840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:10.575799 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:11.374011 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:11.373979 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:12.148644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:12.148581 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:12.850613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:12.850578 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.220965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.220937 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:30:13.329244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329204 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329396 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329482 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329608 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329666 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329726 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329774 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329819 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.329951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.329912 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\") pod \"d1e8413f-16d8-42b3-a5d1-517199912fc0\" (UID: \"d1e8413f-16d8-42b3-a5d1-517199912fc0\") " Nov 03 20:30:13.330175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.330137 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:13.330375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.330350 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:13.330883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.330857 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.330976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.330890 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.332023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.331995 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:13.332351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.332327 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:13.332431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.332370 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:13.333783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.333731 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:13.333783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.333737 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:13.334206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.334181 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "kube-api-access-gkcgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:13.335002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.334974 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy" pod "d1e8413f-16d8-42b3-a5d1-517199912fc0" (UID: "d1e8413f-16d8-42b3-a5d1-517199912fc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:13.432955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.432922 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.432955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.432956 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.433118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.432969 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.433118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.432985 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcgn\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-kube-api-access-gkcgn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.433118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.432999 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1e8413f-16d8-42b3-a5d1-517199912fc0-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.433118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.433013 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1e8413f-16d8-42b3-a5d1-517199912fc0-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.433118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.433025 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1e8413f-16d8-42b3-a5d1-517199912fc0-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:13.443234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.443172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.755189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.755042 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.918128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.917086 5954 generic.go:356] "Generic (PLEG): container finished" podID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerID="6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729" exitCode=0 Nov 03 20:30:13.918128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.917208 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" Nov 03 20:30:13.918128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.917380 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerDied","Data":"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729"} Nov 03 20:30:13.918128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.917424 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f" event={"ID":"d1e8413f-16d8-42b3-a5d1-517199912fc0","Type":"ContainerDied","Data":"3f064bd307514f5788117d40046cbee6981a50b0cb6406f710f461eac7e373df"} Nov 03 20:30:13.918128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.917451 5954 scope.go:122] "RemoveContainer" containerID="6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729" Nov 03 20:30:13.930593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.930566 5954 scope.go:122] "RemoveContainer" containerID="2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1" Nov 03 20:30:13.963204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.963167 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:30:13.967266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.967237 5954 scope.go:122] "RemoveContainer" containerID="6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729" Nov 03 20:30:13.967693 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:13.967661 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729\": container with ID starting with 6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729 not found: ID does not exist" containerID="6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729" Nov 03 20:30:13.967756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.967707 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729"} err="failed to get container status \"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729\": rpc error: code = NotFound desc = could not find container \"6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729\": container with ID starting with 6b1883bb796e55f6d36e094c34ee9f84235e1fc8042bbb46fbb12e4fd9e4e729 not found: ID does not exist" Nov 03 20:30:13.967756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.967733 5954 scope.go:122] "RemoveContainer" containerID="2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1" Nov 03 20:30:13.968595 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:13.968567 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1\": container with ID starting with 2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1 not found: ID does not exist" containerID="2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1" Nov 03 20:30:13.968683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.968603 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1"} err="failed to get container status \"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1\": rpc error: code = NotFound desc = could not find container \"2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1\": container with ID starting with 2a47b744c4e43192b5b282c41c945cc247e52ec6d7bc8d3f8124555dc7c3d6a1 not found: ID does not exist" Nov 03 20:30:13.970589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:13.970561 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-683ba6-openshift-default-84776d9899-shc5f"] Nov 03 20:30:14.499429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:14.499398 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:15.190991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:15.190959 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:15.409006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:15.408978 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" path="/var/lib/kubelet/pods/d1e8413f-16d8-42b3-a5d1-517199912fc0/volumes" Nov 03 20:30:15.940329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:15.940296 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:16.232000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.231907 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:30:16.232473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.232179 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" containerID="cri-o://0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4" gracePeriod=30 Nov 03 20:30:16.237797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.237774 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:30:16.237980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.237960 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" containerID="cri-o://5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a" gracePeriod=30 Nov 03 20:30:16.397816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.397775 5954 scope.go:122] "RemoveContainer" containerID="01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe" Nov 03 20:30:16.397988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.397921 5954 scope.go:122] "RemoveContainer" containerID="1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62" Nov 03 20:30:16.398078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.398057 5954 scope.go:122] "RemoveContainer" containerID="408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de" Nov 03 20:30:16.666076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.666048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:16.945054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.945026 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerStarted","Data":"ccc0266ff835b80f94fd1959bb624251ea742ec66b3c0fe3bcc20e4a8cf7d1c0"} Nov 03 20:30:16.947167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.947139 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerStarted","Data":"3c2f5826274cb208e909d4b0aa8d62d8ed42ffedc5dcbc8dc7475b46b7e09fc8"} Nov 03 20:30:16.948839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:16.948815 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerStarted","Data":"2f9f96cc5f00f3d387b5951bded21650f58d1276b0a8198405b2c0a07f46421c"} Nov 03 20:30:17.375786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.375750 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:17.398165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.398135 5954 scope.go:122] "RemoveContainer" containerID="b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef" Nov 03 20:30:17.958780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.958748 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerStarted","Data":"42a7ec37d46cc709465450b0e6142b69ca941ac14b53e96fb13546da7c858132"} Nov 03 20:30:17.961352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.961325 5954 generic.go:356] "Generic (PLEG): container finished" podID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerID="3c2f5826274cb208e909d4b0aa8d62d8ed42ffedc5dcbc8dc7475b46b7e09fc8" exitCode=60 Nov 03 20:30:17.961477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.961386 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerDied","Data":"3c2f5826274cb208e909d4b0aa8d62d8ed42ffedc5dcbc8dc7475b46b7e09fc8"} Nov 03 20:30:17.961477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.961435 5954 scope.go:122] "RemoveContainer" containerID="408bf93d01df712f7343f0a731b9ba478d6db12474acc9709cd2bca712d054de" Nov 03 20:30:17.961814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.961794 5954 scope.go:122] "RemoveContainer" containerID="3c2f5826274cb208e909d4b0aa8d62d8ed42ffedc5dcbc8dc7475b46b7e09fc8" Nov 03 20:30:17.962298 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:17.962240 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh_odh-ai-gateway-infra(7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" Nov 03 20:30:17.963618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.963592 5954 generic.go:356] "Generic (PLEG): container finished" podID="06832e97-39ca-478b-91af-70eacec17a09" containerID="2f9f96cc5f00f3d387b5951bded21650f58d1276b0a8198405b2c0a07f46421c" exitCode=60 Nov 03 20:30:17.963749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.963655 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerDied","Data":"2f9f96cc5f00f3d387b5951bded21650f58d1276b0a8198405b2c0a07f46421c"} Nov 03 20:30:17.964081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.964061 5954 scope.go:122] "RemoveContainer" containerID="2f9f96cc5f00f3d387b5951bded21650f58d1276b0a8198405b2c0a07f46421c" Nov 03 20:30:17.964464 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:17.964438 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2_odh-ai-gateway-infra(06832e97-39ca-478b-91af-70eacec17a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" podUID="06832e97-39ca-478b-91af-70eacec17a09" Nov 03 20:30:17.965715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.965695 5954 generic.go:356] "Generic (PLEG): container finished" podID="2f838a86-cd37-469a-90e7-47480ce31aff" containerID="ccc0266ff835b80f94fd1959bb624251ea742ec66b3c0fe3bcc20e4a8cf7d1c0" exitCode=60 Nov 03 20:30:17.965824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.965727 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerDied","Data":"ccc0266ff835b80f94fd1959bb624251ea742ec66b3c0fe3bcc20e4a8cf7d1c0"} Nov 03 20:30:17.966048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.966035 5954 scope.go:122] "RemoveContainer" containerID="ccc0266ff835b80f94fd1959bb624251ea742ec66b3c0fe3bcc20e4a8cf7d1c0" Nov 03 20:30:17.966347 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:17.966325 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq_odh-ai-gateway-infra(2f838a86-cd37-469a-90e7-47480ce31aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" Nov 03 20:30:17.980785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.980762 5954 scope.go:122] "RemoveContainer" containerID="1315ba913316637f7095f81fb2ab95563ac36a1553be88b3f2d5dc6c3b8bea62" Nov 03 20:30:17.983521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.983500 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:17.998063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.998039 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:17.998574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:17.998556 5954 scope.go:122] "RemoveContainer" containerID="01f43a61cfea9bf4ae116860688ceab739c9e57e89fa33cf4c4d01c5c2194cbe" Nov 03 20:30:18.012700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.012671 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:18.298140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.298023 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.421898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.420458 5954 scope.go:122] "RemoveContainer" containerID="6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5" Nov 03 20:30:18.441537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.428034 5954 scope.go:122] "RemoveContainer" containerID="00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e" Nov 03 20:30:18.441537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.436996 5954 scope.go:122] "RemoveContainer" containerID="f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e" Nov 03 20:30:18.753822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.753791 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:30:18.840340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.840264 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.946869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.946830 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.992522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.992428 5954 generic.go:356] "Generic (PLEG): container finished" podID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerID="42a7ec37d46cc709465450b0e6142b69ca941ac14b53e96fb13546da7c858132" exitCode=60 Nov 03 20:30:18.992522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.992494 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerDied","Data":"42a7ec37d46cc709465450b0e6142b69ca941ac14b53e96fb13546da7c858132"} Nov 03 20:30:18.992768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.992580 5954 scope.go:122] "RemoveContainer" containerID="b9f04d4a768af7f2b8b782b09cfdba0e3847793c7a810da93903e3c6ffc30bef" Nov 03 20:30:18.993929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.993905 5954 scope.go:122] "RemoveContainer" containerID="42a7ec37d46cc709465450b0e6142b69ca941ac14b53e96fb13546da7c858132" Nov 03 20:30:18.995433 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:18.994380 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx_odh-ai-gateway-infra(f04aa9d5-906e-4480-a073-6c28434e1a5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" Nov 03 20:30:18.995935 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:18.995918 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:19.002399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:19.002373 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerStarted","Data":"53eb0c72322c262bea0716f3e558ad5cc997351f832a4ce3b15dbe9f90b9ff10"} Nov 03 20:30:19.519757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:19.519728 5954 scope.go:122] "RemoveContainer" containerID="aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64" Nov 03 20:30:19.652507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:19.652467 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:20.019410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.019313 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" event={"ID":"2f838a86-cd37-469a-90e7-47480ce31aff","Type":"ContainerDied","Data":"91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20"} Nov 03 20:30:20.019410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.019378 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91cd4cc6c1484b6bb0ae78e168b14c821f27751597564b5479c29c0d1c228c20" Nov 03 20:30:20.022744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.022716 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerStarted","Data":"0976f3e4efe599d695d39b747d94cfa657e69f4054c7669dbed500ba8533c620"} Nov 03 20:30:20.027524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.027496 5954 generic.go:356] "Generic (PLEG): container finished" podID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerID="6f661c3c5a13dc313e0ff568482137e0e75e3c44078b19cd6a866668a965fd6e" exitCode=60 Nov 03 20:30:20.027652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.027531 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerDied","Data":"6f661c3c5a13dc313e0ff568482137e0e75e3c44078b19cd6a866668a965fd6e"} Nov 03 20:30:20.027652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.027589 5954 scope.go:122] "RemoveContainer" containerID="f51524ffd2a0e7d6c9471b8cecbb40a6d532bfd61b7715d2414dbecdb80d060e" Nov 03 20:30:20.028766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.028220 5954 scope.go:122] "RemoveContainer" containerID="6f661c3c5a13dc313e0ff568482137e0e75e3c44078b19cd6a866668a965fd6e" Nov 03 20:30:20.028766 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:20.028701 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q_odh-ai-gateway-infra(17a64efc-6e67-4742-8542-b6b49ea06f98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" Nov 03 20:30:20.033974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.033950 5954 generic.go:356] "Generic (PLEG): container finished" podID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerID="53eb0c72322c262bea0716f3e558ad5cc997351f832a4ce3b15dbe9f90b9ff10" exitCode=60 Nov 03 20:30:20.034081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.034068 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerDied","Data":"53eb0c72322c262bea0716f3e558ad5cc997351f832a4ce3b15dbe9f90b9ff10"} Nov 03 20:30:20.034582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.034532 5954 scope.go:122] "RemoveContainer" containerID="53eb0c72322c262bea0716f3e558ad5cc997351f832a4ce3b15dbe9f90b9ff10" Nov 03 20:30:20.036088 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:20.035884 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-8m8lw_odh-ai-gateway-infra(8a5a9d56-ffd2-4307-832a-30afacf4b48a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" Nov 03 20:30:20.040503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.040400 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" event={"ID":"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269","Type":"ContainerDied","Data":"d16338c4cdd1f0b6a3e72daa8840e152d6b8e3a6e23f901fedd35aff6d2f9a0c"} Nov 03 20:30:20.040503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.040451 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16338c4cdd1f0b6a3e72daa8840e152d6b8e3a6e23f901fedd35aff6d2f9a0c" Nov 03 20:30:20.043751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.043683 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" event={"ID":"06832e97-39ca-478b-91af-70eacec17a09","Type":"ContainerDied","Data":"2bce34eee7c6fbe325a5061b6bb07888df90b486975e806975f75723b1051114"} Nov 03 20:30:20.043751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.043729 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bce34eee7c6fbe325a5061b6bb07888df90b486975e806975f75723b1051114" Nov 03 20:30:20.097212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.096342 5954 scope.go:122] "RemoveContainer" containerID="6ff84fa019db49e98b3a86895432350403c147c4707483247f9a57f502a42de5" Nov 03 20:30:20.109680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.109649 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:20.110478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.110454 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:20.121100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.120882 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:20.125307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.124556 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:20.200612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200574 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\") pod \"2f838a86-cd37-469a-90e7-47480ce31aff\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " Nov 03 20:30:20.200786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200632 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\" (UniqueName: \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\") pod \"06832e97-39ca-478b-91af-70eacec17a09\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " Nov 03 20:30:20.200786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200686 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\") pod \"06832e97-39ca-478b-91af-70eacec17a09\" (UID: \"06832e97-39ca-478b-91af-70eacec17a09\") " Nov 03 20:30:20.200882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200784 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\" (UniqueName: \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\") pod \"2f838a86-cd37-469a-90e7-47480ce31aff\" (UID: \"2f838a86-cd37-469a-90e7-47480ce31aff\") " Nov 03 20:30:20.200931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200911 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\" (UniqueName: \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\") pod \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " Nov 03 20:30:20.201009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.200957 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\") pod \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\" (UID: \"7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269\") " Nov 03 20:30:20.201643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.201612 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca" pod "06832e97-39ca-478b-91af-70eacec17a09" (UID: "06832e97-39ca-478b-91af-70eacec17a09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.202221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.202193 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06832e97-39ca-478b-91af-70eacec17a09-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.202368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.202246 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca" pod "7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" (UID: "7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.202785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.202762 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca" pod "2f838a86-cd37-469a-90e7-47480ce31aff" (UID: "2f838a86-cd37-469a-90e7-47480ce31aff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.213998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.213975 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts" pod "2f838a86-cd37-469a-90e7-47480ce31aff" (UID: "2f838a86-cd37-469a-90e7-47480ce31aff"). InnerVolumeSpecName "kube-api-access-bdrts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.214080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.214050 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7" pod "06832e97-39ca-478b-91af-70eacec17a09" (UID: "06832e97-39ca-478b-91af-70eacec17a09"). InnerVolumeSpecName "kube-api-access-k8sb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.214445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.214426 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz" pod "7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" (UID: "7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269"). InnerVolumeSpecName "kube-api-access-rbrqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.304373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.304291 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f838a86-cd37-469a-90e7-47480ce31aff-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.304373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.304329 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sb7\" (UniqueName: \"kubernetes.io/projected/06832e97-39ca-478b-91af-70eacec17a09-kube-api-access-k8sb7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.304373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.304346 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdrts\" (UniqueName: \"kubernetes.io/projected/2f838a86-cd37-469a-90e7-47480ce31aff-kube-api-access-bdrts\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.304373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.304361 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrqz\" (UniqueName: \"kubernetes.io/projected/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-kube-api-access-rbrqz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.304616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.304376 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.329707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.329678 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:20.405327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.405292 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\" (UniqueName: \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\") pod \"f04aa9d5-906e-4480-a073-6c28434e1a5c\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " Nov 03 20:30:20.405534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.405358 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\") pod \"f04aa9d5-906e-4480-a073-6c28434e1a5c\" (UID: \"f04aa9d5-906e-4480-a073-6c28434e1a5c\") " Nov 03 20:30:20.405740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.405715 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca" pod "f04aa9d5-906e-4480-a073-6c28434e1a5c" (UID: "f04aa9d5-906e-4480-a073-6c28434e1a5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.406283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.406261 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f04aa9d5-906e-4480-a073-6c28434e1a5c-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.407466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.407443 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf" pod "f04aa9d5-906e-4480-a073-6c28434e1a5c" (UID: "f04aa9d5-906e-4480-a073-6c28434e1a5c"). InnerVolumeSpecName "kube-api-access-2p4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.508544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.508507 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4vf\" (UniqueName: \"kubernetes.io/projected/f04aa9d5-906e-4480-a073-6c28434e1a5c-kube-api-access-2p4vf\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.582324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:20.582295 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:21.062644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.062615 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:21.063088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.063065 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerID="bc00dd94fcae038bc69dbadf236115e374f45b99a9f0a8e9c4e009b2c769bc33" exitCode=60 Nov 03 20:30:21.063185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.063155 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerDied","Data":"bc00dd94fcae038bc69dbadf236115e374f45b99a9f0a8e9c4e009b2c769bc33"} Nov 03 20:30:21.063223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.063209 5954 scope.go:122] "RemoveContainer" containerID="aa28cf89fc2cfbc8cb897155600cdc090dbfbc931dc08fa780a104f5fa41ac64" Nov 03 20:30:21.063699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.063679 5954 scope.go:122] "RemoveContainer" containerID="bc00dd94fcae038bc69dbadf236115e374f45b99a9f0a8e9c4e009b2c769bc33" Nov 03 20:30:21.064519 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:21.064139 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9_odh-ai-gateway-infra(ff04b177-e9bb-497e-b644-48e0ac2b758b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" Nov 03 20:30:21.066509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.066487 5954 generic.go:356] "Generic (PLEG): container finished" podID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerID="0976f3e4efe599d695d39b747d94cfa657e69f4054c7669dbed500ba8533c620" exitCode=60 Nov 03 20:30:21.066615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.066554 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerDied","Data":"0976f3e4efe599d695d39b747d94cfa657e69f4054c7669dbed500ba8533c620"} Nov 03 20:30:21.072311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.072289 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" Nov 03 20:30:21.076985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.072431 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2" Nov 03 20:30:21.076985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.072481 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx" event={"ID":"f04aa9d5-906e-4480-a073-6c28434e1a5c","Type":"ContainerDied","Data":"1ca4d8c15d39795fff9cfc797f622c1907314c81c6b7e118b5f169c1dbfe44b6"} Nov 03 20:30:21.076985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.072560 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq" Nov 03 20:30:21.076985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.072942 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh" Nov 03 20:30:21.095645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.095622 5954 scope.go:122] "RemoveContainer" containerID="00d812f60fca850302f138f0cce58c555600d3c6548f843fc79239ce89aa137e" Nov 03 20:30:21.114942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.114735 5954 scope.go:122] "RemoveContainer" containerID="42a7ec37d46cc709465450b0e6142b69ca941ac14b53e96fb13546da7c858132" Nov 03 20:30:21.145888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.145849 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:21.189158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.185723 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:21.199369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.198065 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6cf8ae-29895630-xbhfh"] Nov 03 20:30:21.252073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.252036 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:21.260968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.260939 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7ae0f1-29895630-4btpq"] Nov 03 20:30:21.285919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.285891 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:21.291460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.291431 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3ef4b9-29895630-j49j2"] Nov 03 20:30:21.306221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.306180 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:21.310527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.310500 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ee4c78-29895630-8bsdx"] Nov 03 20:30:21.450982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.450619 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06832e97-39ca-478b-91af-70eacec17a09" path="/var/lib/kubelet/pods/06832e97-39ca-478b-91af-70eacec17a09/volumes" Nov 03 20:30:21.451557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.451530 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" path="/var/lib/kubelet/pods/2f838a86-cd37-469a-90e7-47480ce31aff/volumes" Nov 03 20:30:21.452227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.452202 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" path="/var/lib/kubelet/pods/7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269/volumes" Nov 03 20:30:21.453346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.453321 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" path="/var/lib/kubelet/pods/f04aa9d5-906e-4480-a073-6c28434e1a5c/volumes" Nov 03 20:30:21.453982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.453963 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:21.457241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.457168 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:21.483623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.483588 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:21.537274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537178 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\") pod \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " Nov 03 20:30:21.538617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537510 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\" (UniqueName: \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\") pod \"17a64efc-6e67-4742-8542-b6b49ea06f98\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " Nov 03 20:30:21.538617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537711 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\") pod \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " Nov 03 20:30:21.538617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537754 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\") pod \"17a64efc-6e67-4742-8542-b6b49ea06f98\" (UID: \"17a64efc-6e67-4742-8542-b6b49ea06f98\") " Nov 03 20:30:21.538617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537830 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\" (UniqueName: \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\") pod \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\" (UID: \"8a5a9d56-ffd2-4307-832a-30afacf4b48a\") " Nov 03 20:30:21.538617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.537944 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\" (UniqueName: \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\") pod \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\" (UID: \"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e\") " Nov 03 20:30:21.540180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.540153 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca" pod "8a5a9d56-ffd2-4307-832a-30afacf4b48a" (UID: "8a5a9d56-ffd2-4307-832a-30afacf4b48a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.544944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.544699 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca" pod "17a64efc-6e67-4742-8542-b6b49ea06f98" (UID: "17a64efc-6e67-4742-8542-b6b49ea06f98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.545593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.545235 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca" pod "62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" (UID: "62f42e1a-5464-4d1f-be9d-6e50c12c9c1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.547204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.547168 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd" pod "17a64efc-6e67-4742-8542-b6b49ea06f98" (UID: "17a64efc-6e67-4742-8542-b6b49ea06f98"). InnerVolumeSpecName "kube-api-access-khqhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.548039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.548016 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75" pod "62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" (UID: "62f42e1a-5464-4d1f-be9d-6e50c12c9c1e"). InnerVolumeSpecName "kube-api-access-p2k75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.548662 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.548636 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh" pod "8a5a9d56-ffd2-4307-832a-30afacf4b48a" (UID: "8a5a9d56-ffd2-4307-832a-30afacf4b48a"). InnerVolumeSpecName "kube-api-access-9mzmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.641191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.641099 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5a9d56-ffd2-4307-832a-30afacf4b48a-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.641615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.641592 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqhd\" (UniqueName: \"kubernetes.io/projected/17a64efc-6e67-4742-8542-b6b49ea06f98-kube-api-access-khqhd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.641829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.641757 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.641999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.641963 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a64efc-6e67-4742-8542-b6b49ea06f98-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.642130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.642100 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzmh\" (UniqueName: \"kubernetes.io/projected/8a5a9d56-ffd2-4307-832a-30afacf4b48a-kube-api-access-9mzmh\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.642218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.642207 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2k75\" (UniqueName: \"kubernetes.io/projected/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e-kube-api-access-p2k75\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.705816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.705782 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774356 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774433 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774555 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774600 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774649 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774692 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774767 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774802 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.774972 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\") pod \"9081ec50-cc11-4232-af9c-178cacc31f6b\" (UID: \"9081ec50-cc11-4232-af9c-178cacc31f6b\") " Nov 03 20:30:21.784163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.776950 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.796470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.789065 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.796470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.795855 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.798630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.797313 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.808900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.808291 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.816125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.816076 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:30:21.824560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.824523 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:21.838643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.838598 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.838937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.838912 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:21.839659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.839636 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.842784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.842743 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974" pod "9081ec50-cc11-4232-af9c-178cacc31f6b" (UID: "9081ec50-cc11-4232-af9c-178cacc31f6b"). InnerVolumeSpecName "kube-api-access-95974". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.877421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877384 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877411 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877421 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877423 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877432 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877442 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877451 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877459 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9081ec50-cc11-4232-af9c-178cacc31f6b-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877469 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95974\" (UniqueName: \"kubernetes.io/projected/9081ec50-cc11-4232-af9c-178cacc31f6b-kube-api-access-95974\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.877695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.877479 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9081ec50-cc11-4232-af9c-178cacc31f6b-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.978959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.978874 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.978959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.978939 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.978971 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979031 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979066 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.979180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979100 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979160 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979349 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979418 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979436 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.979572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979475 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.979572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979480 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\") pod \"9ed6641d-b752-4010-b07e-bca3c57fe05e\" (UID: \"9ed6641d-b752-4010-b07e-bca3c57fe05e\") " Nov 03 20:30:21.979805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979780 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.979881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.979790 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.980620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.980593 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.980746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.980627 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.980746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.980643 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.980746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.980658 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.980746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.980672 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed6641d-b752-4010-b07e-bca3c57fe05e-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.981584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.981565 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:21.981669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.981590 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.981839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.981820 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:21.981879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:21.981820 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb" pod "9ed6641d-b752-4010-b07e-bca3c57fe05e" (UID: "9ed6641d-b752-4010-b07e-bca3c57fe05e"). InnerVolumeSpecName "kube-api-access-b6qjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.082033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.081997 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qjb\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-kube-api-access-b6qjb\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.082033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.082033 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.082282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.082044 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.082282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.082053 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed6641d-b752-4010-b07e-bca3c57fe05e-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.083633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.083611 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" Nov 03 20:30:22.083633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.083624 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2" event={"ID":"62f42e1a-5464-4d1f-be9d-6e50c12c9c1e","Type":"ContainerDied","Data":"62cfcec19f2815df5cb388ef0f719717a93bbb5c6b9d5c32df5f2088922add7a"} Nov 03 20:30:22.083788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.083680 5954 scope.go:122] "RemoveContainer" containerID="0976f3e4efe599d695d39b747d94cfa657e69f4054c7669dbed500ba8533c620" Nov 03 20:30:22.085168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.085143 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" Nov 03 20:30:22.085168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.085150 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q" event={"ID":"17a64efc-6e67-4742-8542-b6b49ea06f98","Type":"ContainerDied","Data":"8aa8bd783f0e54b9ad392a78f05816bf08c5d4d44b1194682f823d3f328e935a"} Nov 03 20:30:22.086572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.086552 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" Nov 03 20:30:22.086572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.086570 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw" event={"ID":"8a5a9d56-ffd2-4307-832a-30afacf4b48a","Type":"ContainerDied","Data":"30e01d4aebcb451b395df30edc5660d3d4709f90c282f5614d4acce02da0a2ed"} Nov 03 20:30:22.092635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.092610 5954 generic.go:356] "Generic (PLEG): container finished" podID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerID="0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4" exitCode=0 Nov 03 20:30:22.092821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.092698 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" Nov 03 20:30:22.092821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.092724 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerDied","Data":"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4"} Nov 03 20:30:22.092821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.092773 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv" event={"ID":"9081ec50-cc11-4232-af9c-178cacc31f6b","Type":"ContainerDied","Data":"ac12b16f2a87836e651c0619cb881c2151b132cc4e2ae0232af7f467950ac798"} Nov 03 20:30:22.094879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.094854 5954 generic.go:356] "Generic (PLEG): container finished" podID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerID="5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a" exitCode=0 Nov 03 20:30:22.094971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.094907 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" event={"ID":"9ed6641d-b752-4010-b07e-bca3c57fe05e","Type":"ContainerDied","Data":"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a"} Nov 03 20:30:22.094971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.094946 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" event={"ID":"9ed6641d-b752-4010-b07e-bca3c57fe05e","Type":"ContainerDied","Data":"0c063256fb8d5dd2e3cf438b81295497ab5a381aeecfde9275f2308dea9bcb6b"} Nov 03 20:30:22.095087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.095009 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx" Nov 03 20:30:22.098428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.098414 5954 scope.go:122] "RemoveContainer" containerID="6f661c3c5a13dc313e0ff568482137e0e75e3c44078b19cd6a866668a965fd6e" Nov 03 20:30:22.101635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.101615 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:22.112391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.112377 5954 scope.go:122] "RemoveContainer" containerID="53eb0c72322c262bea0716f3e558ad5cc997351f832a4ce3b15dbe9f90b9ff10" Nov 03 20:30:22.124331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.124308 5954 scope.go:122] "RemoveContainer" containerID="0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4" Nov 03 20:30:22.130329 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.130296 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:22.135185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.135149 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c684d-29895630-drwq2"] Nov 03 20:30:22.143698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.143624 5954 scope.go:122] "RemoveContainer" containerID="4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7" Nov 03 20:30:22.162263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.162234 5954 scope.go:122] "RemoveContainer" containerID="0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4" Nov 03 20:30:22.162662 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:22.162633 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4\": container with ID starting with 0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4 not found: ID does not exist" containerID="0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4" Nov 03 20:30:22.162734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.162678 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4"} err="failed to get container status \"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4\": rpc error: code = NotFound desc = could not find container \"0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4\": container with ID starting with 0a527de5bf36e5509e1db39ee3f08964b082333dd873fd6f5e1c70ab8ea7c5b4 not found: ID does not exist" Nov 03 20:30:22.162734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.162706 5954 scope.go:122] "RemoveContainer" containerID="4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7" Nov 03 20:30:22.162986 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:22.162965 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7\": container with ID starting with 4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7 not found: ID does not exist" containerID="4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7" Nov 03 20:30:22.163044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.162993 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7"} err="failed to get container status \"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7\": rpc error: code = NotFound desc = could not find container \"4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7\": container with ID starting with 4730c861861855cd90f9d0ee87062f736618e18b0c9f0b79f344eb4a0aee53e7 not found: ID does not exist" Nov 03 20:30:22.163044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.163012 5954 scope.go:122] "RemoveContainer" containerID="5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a" Nov 03 20:30:22.175769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.175740 5954 scope.go:122] "RemoveContainer" containerID="5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a" Nov 03 20:30:22.176434 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:22.176403 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a\": container with ID starting with 5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a not found: ID does not exist" containerID="5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a" Nov 03 20:30:22.176546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.176457 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a"} err="failed to get container status \"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a\": rpc error: code = NotFound desc = could not find container \"5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a\": container with ID starting with 5fc7201a3dbf238c2b12507ae18215a618c8b4c29fc0543e79f6f47c14a9ef3a not found: ID does not exist" Nov 03 20:30:22.178291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.178265 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:22.186392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.186351 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-8m8lw"] Nov 03 20:30:22.207914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.207879 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:22.215533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.215504 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895630-hfn5q"] Nov 03 20:30:22.227858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.227831 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:30:22.231573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.231507 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-5hhdx"] Nov 03 20:30:22.241853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.241819 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:30:22.253150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.253102 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-mzhqv"] Nov 03 20:30:22.283153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.283124 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:22.386587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.386556 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\") pod \"ff04b177-e9bb-497e-b644-48e0ac2b758b\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " Nov 03 20:30:22.386793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.386648 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\" (UniqueName: \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\") pod \"ff04b177-e9bb-497e-b644-48e0ac2b758b\" (UID: \"ff04b177-e9bb-497e-b644-48e0ac2b758b\") " Nov 03 20:30:22.386942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.386907 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca" pod "ff04b177-e9bb-497e-b644-48e0ac2b758b" (UID: "ff04b177-e9bb-497e-b644-48e0ac2b758b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:22.387435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.387415 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff04b177-e9bb-497e-b644-48e0ac2b758b-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.388732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.388711 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx" pod "ff04b177-e9bb-497e-b644-48e0ac2b758b" (UID: "ff04b177-e9bb-497e-b644-48e0ac2b758b"). InnerVolumeSpecName "kube-api-access-9kcfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.489652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.489569 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcfx\" (UniqueName: \"kubernetes.io/projected/ff04b177-e9bb-497e-b644-48e0ac2b758b-kube-api-access-9kcfx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.621917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:22.621871 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:23.106487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.106451 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" Nov 03 20:30:23.106908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.106460 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9" event={"ID":"ff04b177-e9bb-497e-b644-48e0ac2b758b","Type":"ContainerDied","Data":"66a57de0ce28a34978144a0a02ebba28813aadf1e29342c04bc2c3a4ec61151c"} Nov 03 20:30:23.106908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.106629 5954 scope.go:122] "RemoveContainer" containerID="bc00dd94fcae038bc69dbadf236115e374f45b99a9f0a8e9c4e009b2c769bc33" Nov 03 20:30:23.131703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.131672 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:23.136244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.136222 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d0898b-29895630-cd9z9"] Nov 03 20:30:23.371594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.371506 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:23.427128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.420895 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" path="/var/lib/kubelet/pods/17a64efc-6e67-4742-8542-b6b49ea06f98/volumes" Nov 03 20:30:23.427128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.421725 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" path="/var/lib/kubelet/pods/62f42e1a-5464-4d1f-be9d-6e50c12c9c1e/volumes" Nov 03 20:30:23.427128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.422418 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" path="/var/lib/kubelet/pods/8a5a9d56-ffd2-4307-832a-30afacf4b48a/volumes" Nov 03 20:30:23.443407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.434286 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" path="/var/lib/kubelet/pods/9081ec50-cc11-4232-af9c-178cacc31f6b/volumes" Nov 03 20:30:23.443407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.435081 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" path="/var/lib/kubelet/pods/9ed6641d-b752-4010-b07e-bca3c57fe05e/volumes" Nov 03 20:30:23.443407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.435728 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" path="/var/lib/kubelet/pods/ff04b177-e9bb-497e-b644-48e0ac2b758b/volumes" Nov 03 20:30:23.754330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:23.754227 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" containerID="cri-o://a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d" gracePeriod=25 Nov 03 20:30:24.266903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:24.266865 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:24.948428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:24.948391 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:25.790938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:25.790885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:26.447295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:26.447263 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:27.164120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:27.164083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:27.903843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:27.903809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:28.588294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:28.588228 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:29.377980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:29.377949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:29.912316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:29.912277 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:30:30.134443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:30.134408 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.310953 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:30.310860 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:30:30.980926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:30.980891 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:31.765269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:31.765222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:32.406433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:32.406392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.088227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:33.088190 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.201198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:33.201147 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.887754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:33.887714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:34.697159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:34.697103 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.465142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:35.465025 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:36.183080 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:36.183049 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:36.889698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:36.889654 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:37.581549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:37.581503 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:38.379700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:38.379670 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.170508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:39.170468 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.880063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:39.880033 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.911049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:39.911006 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:30:40.647521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:40.647472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:41.482695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:41.482660 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:42.176563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:42.176530 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:42.938817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:42.938778 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:43.606005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:43.605972 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:43.838244 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:43.838210 5954 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcfa068_2541_44f6_927c_bc25b97659ed.slice/crio-a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:30:44.064374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.064351 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:30:44.146029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.145936 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\") pod \"2fcfa068-2541-44f6-927c-bc25b97659ed\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " Nov 03 20:30:44.146226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.146064 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\") pod \"2fcfa068-2541-44f6-927c-bc25b97659ed\" (UID: \"2fcfa068-2541-44f6-927c-bc25b97659ed\") " Nov 03 20:30:44.146440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.146415 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume" pod "2fcfa068-2541-44f6-927c-bc25b97659ed" (UID: "2fcfa068-2541-44f6-927c-bc25b97659ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:44.146918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.146895 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcfa068-2541-44f6-927c-bc25b97659ed-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:44.149059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.149038 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6" pod "2fcfa068-2541-44f6-927c-bc25b97659ed" (UID: "2fcfa068-2541-44f6-927c-bc25b97659ed"). InnerVolumeSpecName "kube-api-access-w2rf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:44.248257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.248226 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rf6\" (UniqueName: \"kubernetes.io/projected/2fcfa068-2541-44f6-927c-bc25b97659ed-kube-api-access-w2rf6\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:44.376715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.376680 5954 generic.go:356] "Generic (PLEG): container finished" podID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerID="a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d" exitCode=0 Nov 03 20:30:44.376715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.376717 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" Nov 03 20:30:44.376982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.376761 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" event={"ID":"2fcfa068-2541-44f6-927c-bc25b97659ed","Type":"ContainerDied","Data":"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d"} Nov 03 20:30:44.376982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.376817 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df" event={"ID":"2fcfa068-2541-44f6-927c-bc25b97659ed","Type":"ContainerDied","Data":"b7712cd03ea62b41c8a76bc585f5e1c9131017b7b7fefd2105df4ea7ef233180"} Nov 03 20:30:44.376982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.376841 5954 scope.go:122] "RemoveContainer" containerID="a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d" Nov 03 20:30:44.392737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.392269 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:44.392737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.392701 5954 scope.go:122] "RemoveContainer" containerID="a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d" Nov 03 20:30:44.393189 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:44.393163 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d\": container with ID starting with a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d not found: ID does not exist" containerID="a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d" Nov 03 20:30:44.393267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.393199 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d"} err="failed to get container status \"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d\": rpc error: code = NotFound desc = could not find container \"a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d\": container with ID starting with a3268de5f0029284d74d758a0cd5092ad6b79229b8f8b17eed0b3f1076338b6d not found: ID does not exist" Nov 03 20:30:44.405201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.405148 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:30:44.409053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.409030 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-8013d9-7567bc78fd-r78df"] Nov 03 20:30:44.975599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:44.975543 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:45.150183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:45.150136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:45.534452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:45.534410 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" path="/var/lib/kubelet/pods/2fcfa068-2541-44f6-927c-bc25b97659ed/volumes" Nov 03 20:30:45.539274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:45.539243 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:45.665800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:45.655929 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:46.041886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:46.041808 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:46.444051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:46.444022 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:47.289711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:47.289613 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.000129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:47.999550 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.623331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:48.623285 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.825810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:48.825775 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.998364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:48.996845 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:30:48.998958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:48.998929 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" containerID="cri-o://13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960" gracePeriod=30 Nov 03 20:30:49.680949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:49.680911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:49.785967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:49.785931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:50.127651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:50.127612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:50.673805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:50.673328 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:50.828541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:50.828482 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:30:51.001431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:51.001309 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:51.246740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:51.246708 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:52.121576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:52.121542 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:52.734131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:52.732079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:53.499784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:53.499749 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:53.621694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:53.621615 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:53.981419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:53.981373 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.020748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.020698 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:30:54.021399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.021323 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" containerID="cri-o://23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5" gracePeriod=30 Nov 03 20:30:54.421716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.421674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.813545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.813518 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:30:54.816915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.816888 5954 generic.go:356] "Generic (PLEG): container finished" podID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerID="13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960" exitCode=0 Nov 03 20:30:54.817046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.816931 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" event={"ID":"fea58ac0-9d23-4327-b902-3bd340ebd7f6","Type":"ContainerDied","Data":"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960"} Nov 03 20:30:54.817046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.816977 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" event={"ID":"fea58ac0-9d23-4327-b902-3bd340ebd7f6","Type":"ContainerDied","Data":"4521fb28798601af225c7888ffcc5c50f0d3307c7402bb54c23be69ac2d8c48e"} Nov 03 20:30:54.817046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.817001 5954 scope.go:122] "RemoveContainer" containerID="13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960" Nov 03 20:30:54.834955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.834925 5954 scope.go:122] "RemoveContainer" containerID="13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960" Nov 03 20:30:54.835524 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:30:54.835497 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960\": container with ID starting with 13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960 not found: ID does not exist" containerID="13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960" Nov 03 20:30:54.835609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.835539 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960"} err="failed to get container status \"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960\": rpc error: code = NotFound desc = could not find container \"13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960\": container with ID starting with 13200925bcbcd3bcd87da783a6b201b533e5a904da1210b98fc5901785176960 not found: ID does not exist" Nov 03 20:30:54.944716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.944673 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.944933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.944881 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945039 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945171 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945213 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945257 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945289 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945353 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945405 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\") pod \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\" (UID: \"fea58ac0-9d23-4327-b902-3bd340ebd7f6\") " Nov 03 20:30:54.945621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945591 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.945847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945822 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.945941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.945842 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.946096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946069 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.946244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946186 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:54.946518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946494 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.946613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946524 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.946613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946555 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.946613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946571 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.946613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.946585 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.947759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.947716 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.947897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.947866 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.948710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.948683 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:54.949548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:54.949522 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c" pod "fea58ac0-9d23-4327-b902-3bd340ebd7f6" (UID: "fea58ac0-9d23-4327-b902-3bd340ebd7f6"). InnerVolumeSpecName "kube-api-access-j4z8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:55.048444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.048360 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.048444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.048398 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.048444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.048415 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4z8c\" (UniqueName: \"kubernetes.io/projected/fea58ac0-9d23-4327-b902-3bd340ebd7f6-kube-api-access-j4z8c\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.048444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.048432 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea58ac0-9d23-4327-b902-3bd340ebd7f6-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.351518 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.351475 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:55.826501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.826414 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55" Nov 03 20:30:55.853084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.853050 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:30:55.858911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:55.858889 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-44v55"] Nov 03 20:30:56.278330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:56.278294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.095130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:57.088674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.407937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:57.407901 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" path="/var/lib/kubelet/pods/fea58ac0-9d23-4327-b902-3bd340ebd7f6/volumes" Nov 03 20:30:57.785384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:57.785308 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.957829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:57.957795 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:58.069506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:58.069395 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:58.684785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:58.684751 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:59.386245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:59.386207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:59.489424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:30:59.489394 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:00.206397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.206364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:00.372374 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:31:00.371827 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:31:00.557315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.557291 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:31:00.606265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606229 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606305 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606343 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606383 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606431 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606505 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606618 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.606788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606771 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.607007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606869 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\") pod \"15b7fecc-5c4d-44be-adb0-30654c997e70\" (UID: \"15b7fecc-5c4d-44be-adb0-30654c997e70\") " Nov 03 20:31:00.607007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.606966 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:00.607255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.607233 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.607465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.607442 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.607465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.607456 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.607663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.607639 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.608414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608387 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b7fecc-5c4d-44be-adb0-30654c997e70-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.608563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608417 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.608563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608433 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.608563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608447 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.608563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608461 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.608942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.608916 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.610452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.610427 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:00.610727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.610701 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.611428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.611401 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj" pod "15b7fecc-5c4d-44be-adb0-30654c997e70" (UID: "15b7fecc-5c4d-44be-adb0-30654c997e70"). InnerVolumeSpecName "kube-api-access-mb6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.709847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.709771 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.709847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.709801 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.709847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.709813 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb6xj\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-kube-api-access-mb6xj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.709847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.709823 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b7fecc-5c4d-44be-adb0-30654c997e70-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.907690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.907657 5954 generic.go:356] "Generic (PLEG): container finished" podID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerID="23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5" exitCode=0 Nov 03 20:31:00.907867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.907715 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" Nov 03 20:31:00.907867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.907719 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerDied","Data":"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5"} Nov 03 20:31:00.907867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.907854 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht" event={"ID":"15b7fecc-5c4d-44be-adb0-30654c997e70","Type":"ContainerDied","Data":"edd04eca3eaa6614083ddcade7c0b403de1ba57ccd59fe062a33514c80d3e916"} Nov 03 20:31:00.908032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.907876 5954 scope.go:122] "RemoveContainer" containerID="23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5" Nov 03 20:31:00.920371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.920354 5954 scope.go:122] "RemoveContainer" containerID="8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc" Nov 03 20:31:00.930226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.930078 5954 scope.go:122] "RemoveContainer" containerID="23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5" Nov 03 20:31:00.931080 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:31:00.931059 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5\": container with ID starting with 23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5 not found: ID does not exist" containerID="23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5" Nov 03 20:31:00.931187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.931087 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5"} err="failed to get container status \"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5\": rpc error: code = NotFound desc = could not find container \"23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5\": container with ID starting with 23abc9d91040e9f841debbcc082b83e82cbd58acef36eba7f8658f1c202272c5 not found: ID does not exist" Nov 03 20:31:00.931187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.931117 5954 scope.go:122] "RemoveContainer" containerID="8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc" Nov 03 20:31:00.931440 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:31:00.931412 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc\": container with ID starting with 8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc not found: ID does not exist" containerID="8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc" Nov 03 20:31:00.931497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.931451 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc"} err="failed to get container status \"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc\": rpc error: code = NotFound desc = could not find container \"8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc\": container with ID starting with 8fec610fe12836fe87bbacea7dde60eec313ca21945da270d13ec9711d2a7fcc not found: ID does not exist" Nov 03 20:31:00.935230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.935203 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:31:00.938816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:00.938795 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-mphht"] Nov 03 20:31:01.077748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:01.077674 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:01.407277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:01.407246 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" path="/var/lib/kubelet/pods/15b7fecc-5c4d-44be-adb0-30654c997e70/volumes" Nov 03 20:31:01.801312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:01.801222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:02.701546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:02.701507 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.306678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:03.306648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.424272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:03.424229 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.529273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:03.529244 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.198530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:04.198499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.003255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:05.003223 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.691079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:05.691044 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:06.405381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:06.405315 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:07.199546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:07.199502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:07.889799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:07.889762 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:08.705871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:08.705835 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:09.408527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:09.408486 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.131367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:10.131333 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.239899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:10.239860 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.811140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:10.811079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:11.579509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:11.579473 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:12.276780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:12.276746 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:13.077145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:13.077086 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:13.179372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:13.179331 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:13.967419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:13.967383 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.767401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:14.767364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:15.463554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:15.463519 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:16.205795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.205742 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:16.359571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.359529 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:31:16.361031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.360933 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:31:16.361312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361277 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:31:16.361466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361448 5954 cpu_manager.go:414] "containerMap: removing container" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.361583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361569 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.361705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361692 5954 cpu_manager.go:414] "containerMap: removing container" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.361824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361810 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.361952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.361941 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" Nov 03 20:31:16.362069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362057 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" Nov 03 20:31:16.362192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362180 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.362287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362275 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.362395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362385 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.362500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362486 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.362606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362595 5954 cpu_manager.go:414] "containerMap: removing container" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.362704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362690 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.362804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362793 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.362889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362870 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.362988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.362978 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.363063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363053 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.363177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363165 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" Nov 03 20:31:16.363266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363256 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" Nov 03 20:31:16.363361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363350 5954 cpu_manager.go:414] "containerMap: removing container" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.363475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363463 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.363582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363572 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.363658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363648 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.363752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363742 5954 cpu_manager.go:414] "containerMap: removing container" podUID="de8d6ef3-890b-4146-95a1-f9367f7194b0" containerName="maas-api" Nov 03 20:31:16.363841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363832 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8d6ef3-890b-4146-95a1-f9367f7194b0" containerName="maas-api" Nov 03 20:31:16.363922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363914 5954 cpu_manager.go:414] "containerMap: removing container" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" Nov 03 20:31:16.364001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.363991 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" Nov 03 20:31:16.364098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364089 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.364190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364180 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.364282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364271 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.364372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364362 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.364455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364446 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.364542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364530 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.364631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364622 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.364700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364690 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.364804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364793 5954 cpu_manager.go:414] "containerMap: removing container" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:31:16.364893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364883 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:31:16.364987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.364978 5954 cpu_manager.go:414] "containerMap: removing container" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:31:16.365075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365062 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:31:16.365214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365198 5954 cpu_manager.go:414] "containerMap: removing container" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.365336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365323 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.365442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365432 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.365521 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365511 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.365622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365611 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.365724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365714 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.365809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365800 5954 cpu_manager.go:414] "containerMap: removing container" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.365892 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365879 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.365998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.365986 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.366086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366076 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.366206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366193 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.366308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366297 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.366462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366452 5954 cpu_manager.go:414] "containerMap: removing container" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.366577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366551 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.366686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366676 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.366782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.366772 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.367141 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.367124 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:31:16.367283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.367268 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.367406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.367395 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.367571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.367559 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:31:16.367754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.367743 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.368374 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368358 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.368543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368531 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:31:16.368667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368656 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.368756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368746 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.368834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368825 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:31:16.368912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368904 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.368994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.368984 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:31:16.369071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369062 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.369185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369173 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:31:16.369283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369273 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcfa068-2541-44f6-927c-bc25b97659ed" containerName="payload-processing" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369355 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369366 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369376 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369387 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369397 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed6641d-b752-4010-b07e-bca3c57fe05e" containerName="istio-proxy" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369412 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369423 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8d6ef3-890b-4146-95a1-f9367f7194b0" containerName="maas-api" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369432 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea58ac0-9d23-4327-b902-3bd340ebd7f6" containerName="istio-proxy" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369442 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369452 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9081ec50-cc11-4232-af9c-178cacc31f6b" containerName="istio-proxy" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369461 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.370708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.369471 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f838a86-cd37-469a-90e7-47480ce31aff" containerName="cleanup" Nov 03 20:31:16.376860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.376831 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.377135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.377095 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:31:16.550744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.550655 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.550895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.550776 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.550895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.550871 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.550995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.550904 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmwd\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.551033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.550995 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.551120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.551091 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.551172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.551160 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.551358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.551339 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.551416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.551369 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653017 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653100 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653146 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653337 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653365 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653476 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653537 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653573 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653644 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653671 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmwd\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653777 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.653915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653884 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.654030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.653947 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.654153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.654136 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.655812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.655790 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.656018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.655999 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.665267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.665247 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.665495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.665473 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmwd\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.701074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.701049 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:16.993097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.993066 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:31:16.993746 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:31:16.993722 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba8f2d_957e_4988_8483_1a3e579285c2.slice/crio-5ad8772dfd22445809763065bce383edd77fc7a34c992fbc63176609ff541d97 WatchSource:0}: Error finding container 5ad8772dfd22445809763065bce383edd77fc7a34c992fbc63176609ff541d97: Status 404 returned error can't find the container with id 5ad8772dfd22445809763065bce383edd77fc7a34c992fbc63176609ff541d97 Nov 03 20:31:16.996842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.996809 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:31:16.996960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.996877 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:31:16.996960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:16.996906 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:31:17.001512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:17.001480 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:17.135036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:17.134913 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" event={"ID":"9bba8f2d-957e-4988-8483-1a3e579285c2","Type":"ContainerStarted","Data":"5ad8772dfd22445809763065bce383edd77fc7a34c992fbc63176609ff541d97"} Nov 03 20:31:17.804387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:17.804345 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:18.148766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.148713 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" event={"ID":"9bba8f2d-957e-4988-8483-1a3e579285c2","Type":"ContainerStarted","Data":"31b340494a692fca9dbceed420d9a11d79484aa1deb41e46ddb9569ef5c75caa"} Nov 03 20:31:18.179439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.179389 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podStartSLOduration=2.17937427 podStartE2EDuration="2.17937427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:18.175812604 +0000 UTC m=+2018.900820201" watchObservedRunningTime="2026-11-03 20:31:18.17937427 +0000 UTC m=+2018.904381866" Nov 03 20:31:18.478318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.478229 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:18.701734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.701699 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:18.703232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.703206 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:31:18.703369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:18.703253 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 20:31:19.235705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:19.235593 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:19.701639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:19.701604 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:31:19.701821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:19.701662 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 20:31:20.198913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:20.198878 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:20.702205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:20.702171 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:31:20.702636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:20.702226 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 20:31:20.886920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:20.886885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:21.644796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:21.644763 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:21.709185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:21.707060 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:31:21.709185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:21.707143 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 20:31:22.308535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:22.308500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:22.702190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:22.702153 5954 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 03 20:31:22.702380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:22.702210 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 03 20:31:23.041577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:23.041388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:23.707017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:23.706987 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:23.707485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:23.707464 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:23.708205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:23.708185 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:31:23.790576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:23.790537 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:24.490298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:24.490228 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:25.187211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:25.187176 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:25.993882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:25.993847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:26.740555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:26.740525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:27.552762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:27.552729 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:28.328588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:28.328558 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:29.129203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:29.129168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:30.041325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:30.041290 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:30.165127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:30.164914 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:30.409540 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:31:30.409503 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:31:30.991852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:30.991815 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:31.676648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:31.676600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:32.482915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:32.482880 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:33.273057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:33.273027 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:34.214486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:34.214447 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:35.077576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:35.077541 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:35.794922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:35.794889 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:36.476430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:36.476392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:37.261200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:37.261168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:38.025487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:38.025456 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:38.813827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:38.813790 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:39.597806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:39.597770 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:40.372337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:40.372297 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:41.178863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:41.178827 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:41.760739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:41.760701 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:42.252843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:42.252807 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.066799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:43.066771 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.907987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:43.907953 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:44.636587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:44.636545 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:45.495989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:45.495949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:46.767764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:46.767711 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:47.683181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:47.683099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:48.478159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:48.478093 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:48.687463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:48.687423 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:49.373130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:49.372392 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:50.205441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:50.205407 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:50.981275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:50.981235 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:51.783889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:51.783846 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.286252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:52.286215 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.389882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:52.389847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.621493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:52.621449 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:53.681287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:53.681235 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.492677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:54.492639 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:55.206065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:55.206027 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.067750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:56.067718 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.782491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:56.782457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:57.579940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:57.579909 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.298717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:58.298682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.978267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:58.978230 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.711837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:31:59.711804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:00.260547 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:00.260504 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:32:00.468676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:00.468640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.167567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:01.167533 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.911874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:01.911841 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.684425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:02.684385 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:03.376162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:03.376094 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.201330 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:04.201296 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.976771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:04.976737 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.671383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:05.671346 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.398775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:06.398727 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.081946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:07.081910 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.784034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:07.783998 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:08.600997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:08.600957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.335575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:09.335526 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.927552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:09.927516 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.114406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:10.114370 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.234115 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:10.233961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.615024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:10.614987 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.985493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:10.985353 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.488277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:11.488238 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:12.394629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:12.394578 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:13.169914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:13.169875 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:13.894810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:13.894772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:14.668766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:14.668714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:15.377690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:15.377659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:16.185922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:16.185887 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:16.893227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:16.893194 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:17.587687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:17.587649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:17.753151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:17.751064 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.059087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:18.059049 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:32:18.059538 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:18.059284 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" containerID="cri-o://3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c" gracePeriod=30 Nov 03 20:32:18.099389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:18.099336 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:18.880362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:18.880329 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:19.019021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:19.018980 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:32:19.019292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:19.019247 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" containerID="cri-o://41bb2dff1c83efcb984bdc79c1c5070f07c49160f4b66b285d29d9e7a07ed129" gracePeriod=30 Nov 03 20:32:19.787601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:19.787564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.328738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:20.328685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.704026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:20.703974 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.580714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:21.580681 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.315343 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:22.315312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.504318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:22.504283 5954 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:22.504509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:22.504353 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:23.076255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.076218 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:23.442128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.438853 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:32:23.474592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.474553 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:23.603137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.594222 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.623188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.603498 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.661339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.661309 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.661940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.661917 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.662428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.662409 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.662973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.662956 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.663488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.663469 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.663973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.663956 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.664495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.664478 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\") pod \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\" (UID: \"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee\") " Nov 03 20:32:23.665913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.661797 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.666053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.662296 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.666202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.663844 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.666307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.664379 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:23.666403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.664855 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.735377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.735318 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.767130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.766412 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782804 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782840 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782854 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782866 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782881 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782893 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.784133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.782905 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.798032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.795432 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:23.802697 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.802630 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g" pod "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" (UID: "33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee"). InnerVolumeSpecName "kube-api-access-vh99g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.884444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.884365 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.884444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:23.884395 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh99g\" (UniqueName: \"kubernetes.io/projected/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee-kube-api-access-vh99g\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.105137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.102334 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.250208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.249947 5954 generic.go:356] "Generic (PLEG): container finished" podID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerID="41bb2dff1c83efcb984bdc79c1c5070f07c49160f4b66b285d29d9e7a07ed129" exitCode=0 Nov 03 20:32:24.250208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.250023 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" event={"ID":"74ea9a19-fb71-4628-804a-092f6cea7f7e","Type":"ContainerDied","Data":"41bb2dff1c83efcb984bdc79c1c5070f07c49160f4b66b285d29d9e7a07ed129"} Nov 03 20:32:24.255159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.252451 5954 generic.go:356] "Generic (PLEG): container finished" podID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerID="3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c" exitCode=0 Nov 03 20:32:24.255159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.252516 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerDied","Data":"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c"} Nov 03 20:32:24.255159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.252554 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" event={"ID":"33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee","Type":"ContainerDied","Data":"499694a301cc8dfcf81f976d6a6eaafa87fc837b9b0f61370da9d73a0e9077c8"} Nov 03 20:32:24.255159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.252574 5954 scope.go:122] "RemoveContainer" containerID="3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c" Nov 03 20:32:24.255159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.252711 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk" Nov 03 20:32:24.266701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.266679 5954 scope.go:122] "RemoveContainer" containerID="c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615" Nov 03 20:32:24.280210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.280090 5954 scope.go:122] "RemoveContainer" containerID="3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c" Nov 03 20:32:24.282117 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:24.280694 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c\": container with ID starting with 3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c not found: ID does not exist" containerID="3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c" Nov 03 20:32:24.282117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.280766 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c"} err="failed to get container status \"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c\": rpc error: code = NotFound desc = could not find container \"3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c\": container with ID starting with 3c89cbe145ab9de7dcbc861b93389c96b3e9a23539f35ad73024556537e6d03c not found: ID does not exist" Nov 03 20:32:24.282117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.280795 5954 scope.go:122] "RemoveContainer" containerID="c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615" Nov 03 20:32:24.282595 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:24.282572 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615\": container with ID starting with c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615 not found: ID does not exist" containerID="c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615" Nov 03 20:32:24.282689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.282601 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615"} err="failed to get container status \"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615\": rpc error: code = NotFound desc = could not find container \"c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615\": container with ID starting with c511873b06872d656eaafeaa73f09dd684852a4ab8d0a665551e5bc57448c615 not found: ID does not exist" Nov 03 20:32:24.284717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.284574 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:32:24.289300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.289277 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-2fnbk"] Nov 03 20:32:24.433615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.433546 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:32:24.509829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.509791 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.509859 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.509911 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510124 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510171 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510258 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510326 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510694 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510762 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\") pod \"74ea9a19-fb71-4628-804a-092f6cea7f7e\" (UID: \"74ea9a19-fb71-4628-804a-092f6cea7f7e\") " Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510338 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510526 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:24.510876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.510779 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:24.511480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.511101 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:24.511748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.511724 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.511840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.511752 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.511840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.511770 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.511840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.511784 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ea9a19-fb71-4628-804a-092f6cea7f7e-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.512227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.512198 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:24.513850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.513829 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:24.514543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.514512 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "kube-api-access-nftqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:24.514641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.514562 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:24.514641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.514572 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy" pod "74ea9a19-fb71-4628-804a-092f6cea7f7e" (UID: "74ea9a19-fb71-4628-804a-092f6cea7f7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:24.613958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.613921 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.613958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.613952 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.613958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.613961 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.614255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.613970 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftqs\" (UniqueName: \"kubernetes.io/projected/74ea9a19-fb71-4628-804a-092f6cea7f7e-kube-api-access-nftqs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.614255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:24.613980 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ea9a19-fb71-4628-804a-092f6cea7f7e-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:25.072133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.069048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:25.262897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.262865 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" Nov 03 20:32:25.262897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.262883 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd" event={"ID":"74ea9a19-fb71-4628-804a-092f6cea7f7e","Type":"ContainerDied","Data":"e87f3d3782cbe705fc9c3f2ac4f2c6e7663db05ecabfac63733c5dcaa27f1dad"} Nov 03 20:32:25.263419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.262946 5954 scope.go:122] "RemoveContainer" containerID="41bb2dff1c83efcb984bdc79c1c5070f07c49160f4b66b285d29d9e7a07ed129" Nov 03 20:32:25.288504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.288476 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:32:25.294824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.294800 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-fqpfd"] Nov 03 20:32:25.406901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.406866 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" path="/var/lib/kubelet/pods/33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee/volumes" Nov 03 20:32:25.407553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.407523 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" path="/var/lib/kubelet/pods/74ea9a19-fb71-4628-804a-092f6cea7f7e/volumes" Nov 03 20:32:25.885240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:25.885194 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:26.628378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:26.628341 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:27.383718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:27.383684 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.209666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:28.209634 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.798437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:28.798402 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.893399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:28.893367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:29.238185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:29.236745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.061286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:30.061254 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.260515 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:30.260479 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:32:30.635670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:30.635639 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:31.016865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:31.016757 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:31.896413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:31.896375 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.807255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:32.807223 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:33.251654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:33.251609 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:32:33.691240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:33.691185 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:34.733672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:34.733633 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.558603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:35.558567 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:36.368705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:36.368670 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.378565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:37.378527 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:38.131145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:38.131074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:39.147967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:39.147930 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:39.399539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:39.399438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.224823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:40.224790 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.982467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:40.982425 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.792422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:41.792387 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.891220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:41.891176 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.690611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.690578 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:32:42.690863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.690837 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" containerID="cri-o://d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d" gracePeriod=30 Nov 03 20:32:42.699193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.698712 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:32:42.699609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.699554 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" containerID="cri-o://0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd" gracePeriod=30 Nov 03 20:32:42.707563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.707532 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:32:42.708062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.708015 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" containerID="cri-o://524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b" gracePeriod=30 Nov 03 20:32:42.755018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.754956 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:32:42.755898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.755844 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" podUID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" containerName="maas-api" containerID="cri-o://0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96" gracePeriod=30 Nov 03 20:32:42.793854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:42.793810 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.391908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.390492 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.416367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.416336 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:32:43.452844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.452806 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\") pod \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " Nov 03 20:32:43.453032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.452971 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\") pod \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " Nov 03 20:32:43.453085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.453071 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\") pod \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " Nov 03 20:32:43.453184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.453152 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\" (UniqueName: \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\") pod \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\" (UID: \"840416a8-b1bd-4230-83a4-d6960dbfb0d4\") " Nov 03 20:32:43.453900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.453868 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca" pod "840416a8-b1bd-4230-83a4-d6960dbfb0d4" (UID: "840416a8-b1bd-4230-83a4-d6960dbfb0d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:43.455889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.455862 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs" pod "840416a8-b1bd-4230-83a4-d6960dbfb0d4" (UID: "840416a8-b1bd-4230-83a4-d6960dbfb0d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:43.456365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.456337 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls" pod "840416a8-b1bd-4230-83a4-d6960dbfb0d4" (UID: "840416a8-b1bd-4230-83a4-d6960dbfb0d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:43.456477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.456394 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq" pod "840416a8-b1bd-4230-83a4-d6960dbfb0d4" (UID: "840416a8-b1bd-4230-83a4-d6960dbfb0d4"). InnerVolumeSpecName "kube-api-access-85wxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:43.485166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.485073 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.581914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.577715 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:43.581914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.577752 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/840416a8-b1bd-4230-83a4-d6960dbfb0d4-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:43.581914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.577769 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840416a8-b1bd-4230-83a4-d6960dbfb0d4-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:43.581914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.577785 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wxq\" (UniqueName: \"kubernetes.io/projected/840416a8-b1bd-4230-83a4-d6960dbfb0d4-kube-api-access-85wxq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:43.594833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.585608 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.633837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.611737 5954 generic.go:356] "Generic (PLEG): container finished" podID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" containerID="0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96" exitCode=0 Nov 03 20:32:43.633837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.612061 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" event={"ID":"840416a8-b1bd-4230-83a4-d6960dbfb0d4","Type":"ContainerDied","Data":"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96"} Nov 03 20:32:43.633837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.612143 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" event={"ID":"840416a8-b1bd-4230-83a4-d6960dbfb0d4","Type":"ContainerDied","Data":"1824a48f34c152c63826f2aecb516543b75956668cec7b88e3ac43dbca9e04e1"} Nov 03 20:32:43.633837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.612177 5954 scope.go:122] "RemoveContainer" containerID="0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96" Nov 03 20:32:43.633837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.612371 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn" Nov 03 20:32:43.655303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.655272 5954 scope.go:122] "RemoveContainer" containerID="0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96" Nov 03 20:32:43.655699 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:43.655674 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96\": container with ID starting with 0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96 not found: ID does not exist" containerID="0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96" Nov 03 20:32:43.655790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.655708 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96"} err="failed to get container status \"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96\": rpc error: code = NotFound desc = could not find container \"0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96\": container with ID starting with 0e69fe483920f9f26111ee5065ee352da5179b199764ebf1baeae726863dcd96 not found: ID does not exist" Nov 03 20:32:43.657694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.657671 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:32:43.662283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.662259 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-34e844-9cb9d845-9qnqn"] Nov 03 20:32:43.995567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:43.995531 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:44.464050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:44.464017 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:45.406434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:45.406402 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" path="/var/lib/kubelet/pods/840416a8-b1bd-4230-83a4-d6960dbfb0d4/volumes" Nov 03 20:32:45.419638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:45.419573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:45.522597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:45.522560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:46.576062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:46.576019 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:47.628835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:47.628792 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.034427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.034397 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:32:48.115822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.115794 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:32:48.123070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.123037 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:32:48.185753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185664 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185807 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185863 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185906 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185956 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.185987 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.186047 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.186205 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.186341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.186315 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\") pod \"40c47013-4463-4edd-b130-c4b93fea7af9\" (UID: \"40c47013-4463-4edd-b130-c4b93fea7af9\") " Nov 03 20:32:48.187941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.187852 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.187941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.187873 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.187941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.187934 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.188164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.187992 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:48.188627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.188591 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.190589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.190554 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "kube-api-access-9sxct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.190699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.190670 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.190944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.190922 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.191176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.191156 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo" pod "40c47013-4463-4edd-b130-c4b93fea7af9" (UID: "40c47013-4463-4edd-b130-c4b93fea7af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:48.292299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292257 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.292473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292315 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.292667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292640 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.292746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292678 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.292746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292701 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.292869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292746 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.292869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292821 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.292982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292874 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.292982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292906 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.292982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292942 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.292982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.292944 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.293265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293042 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.293265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293182 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.293265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293201 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.293265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293236 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.293489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293281 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.293489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293368 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.293489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293403 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.293489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293442 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.293704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293491 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\") pod \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\" (UID: \"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0\") " Nov 03 20:32:48.293704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293519 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\") pod \"605626b1-de96-4f7e-83de-02621bca7bbc\" (UID: \"605626b1-de96-4f7e-83de-02621bca7bbc\") " Nov 03 20:32:48.293704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.293637 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294183 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294429 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294444 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294627 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294791 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.294997 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295550 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295574 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c47013-4463-4edd-b130-c4b93fea7af9-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295590 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295605 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295618 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605626b1-de96-4f7e-83de-02621bca7bbc-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295631 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c47013-4463-4edd-b130-c4b93fea7af9-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295644 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295657 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295670 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295684 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxct\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-kube-api-access-9sxct\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295698 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295711 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295724 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295737 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295754 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295767 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295790 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c47013-4463-4edd-b130-c4b93fea7af9-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295807 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295822 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c47013-4463-4edd-b130-c4b93fea7af9-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.294666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.295897 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "kube-api-access-thklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.298623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.298559 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "kube-api-access-jb2c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.298623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.298596 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.298795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.298740 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.298841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.298821 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:48.298890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.298871 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:48.299307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.299285 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo" pod "605626b1-de96-4f7e-83de-02621bca7bbc" (UID: "605626b1-de96-4f7e-83de-02621bca7bbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:48.299526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.299509 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy" pod "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" (UID: "0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:48.397883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397851 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.397883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397878 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605626b1-de96-4f7e-83de-02621bca7bbc-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.397883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397888 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.398187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397899 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thklp\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-kube-api-access-thklp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.398187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397909 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2c2\" (UniqueName: \"kubernetes.io/projected/605626b1-de96-4f7e-83de-02621bca7bbc-kube-api-access-jb2c2\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.398187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397919 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.398187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397928 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.398187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.397936 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605626b1-de96-4f7e-83de-02621bca7bbc-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:48.471648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.471567 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.623675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.623445 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.720028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.719990 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.762773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.762645 5954 generic.go:356] "Generic (PLEG): container finished" podID="605626b1-de96-4f7e-83de-02621bca7bbc" containerID="524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b" exitCode=0 Nov 03 20:32:48.762773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.762742 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" Nov 03 20:32:48.763082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.762822 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" event={"ID":"605626b1-de96-4f7e-83de-02621bca7bbc","Type":"ContainerDied","Data":"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b"} Nov 03 20:32:48.763082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.762870 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp" event={"ID":"605626b1-de96-4f7e-83de-02621bca7bbc","Type":"ContainerDied","Data":"91183893c98bb157a75105b561f46e24262da2b6cc582b1d86a3d5bf73410d80"} Nov 03 20:32:48.763082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.762898 5954 scope.go:122] "RemoveContainer" containerID="524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b" Nov 03 20:32:48.764962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.764939 5954 generic.go:356] "Generic (PLEG): container finished" podID="40c47013-4463-4edd-b130-c4b93fea7af9" containerID="d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d" exitCode=0 Nov 03 20:32:48.765063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.764995 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" Nov 03 20:32:48.765063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.765029 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" event={"ID":"40c47013-4463-4edd-b130-c4b93fea7af9","Type":"ContainerDied","Data":"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d"} Nov 03 20:32:48.765213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.765087 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz" event={"ID":"40c47013-4463-4edd-b130-c4b93fea7af9","Type":"ContainerDied","Data":"1ef88e4b6ef8e031ea5c2ec6d4735cc5ea178f6b32bc9fd1cf082b6aced3193a"} Nov 03 20:32:48.770161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.769322 5954 generic.go:356] "Generic (PLEG): container finished" podID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerID="0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd" exitCode=0 Nov 03 20:32:48.770161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.769371 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" event={"ID":"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0","Type":"ContainerDied","Data":"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd"} Nov 03 20:32:48.770161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.769422 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" event={"ID":"0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0","Type":"ContainerDied","Data":"11a190f968287562449f031c2be4a6b1c575c52805a3a943bb459685bf0ec05f"} Nov 03 20:32:48.770161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.769370 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v" Nov 03 20:32:48.776841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.776821 5954 scope.go:122] "RemoveContainer" containerID="524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b" Nov 03 20:32:48.777175 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:48.777153 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b\": container with ID starting with 524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b not found: ID does not exist" containerID="524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b" Nov 03 20:32:48.777260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.777191 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b"} err="failed to get container status \"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b\": rpc error: code = NotFound desc = could not find container \"524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b\": container with ID starting with 524f52a53022b1e8f0885d9d05377b96401a5bd365cb85ceeb16e5cd37dc556b not found: ID does not exist" Nov 03 20:32:48.777260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.777213 5954 scope.go:122] "RemoveContainer" containerID="d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d" Nov 03 20:32:48.807984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.807957 5954 scope.go:122] "RemoveContainer" containerID="d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d" Nov 03 20:32:48.808352 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:48.808332 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d\": container with ID starting with d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d not found: ID does not exist" containerID="d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d" Nov 03 20:32:48.808422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.808361 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d"} err="failed to get container status \"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d\": rpc error: code = NotFound desc = could not find container \"d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d\": container with ID starting with d2c4746dedbf4217f1b87695741fcbaff0217bc41f4214a111b384ba16f1e46d not found: ID does not exist" Nov 03 20:32:48.808422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.808380 5954 scope.go:122] "RemoveContainer" containerID="0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd" Nov 03 20:32:48.880256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.879552 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:32:48.951322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.951288 5954 scope.go:122] "RemoveContainer" containerID="0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd" Nov 03 20:32:48.959128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.952318 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.959887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.959812 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-khb7v"] Nov 03 20:32:48.967293 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:48.967257 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd\": container with ID starting with 0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd not found: ID does not exist" containerID="0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd" Nov 03 20:32:48.967528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.967498 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd"} err="failed to get container status \"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd\": rpc error: code = NotFound desc = could not find container \"0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd\": container with ID starting with 0c1cb72725e522893a991d8bdc58ae30bb9122cb7c68ada716c12945001d10fd not found: ID does not exist" Nov 03 20:32:49.001135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:48.992180 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:32:49.026260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.026162 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gl4rp"] Nov 03 20:32:49.059987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.059249 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:32:49.062986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.062937 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-29vkz"] Nov 03 20:32:49.380246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.380214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:49.408539 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.408505 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" path="/var/lib/kubelet/pods/0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0/volumes" Nov 03 20:32:49.409082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.409064 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" path="/var/lib/kubelet/pods/40c47013-4463-4edd-b130-c4b93fea7af9/volumes" Nov 03 20:32:49.409575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:49.409561 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" path="/var/lib/kubelet/pods/605626b1-de96-4f7e-83de-02621bca7bbc/volumes" Nov 03 20:32:50.170475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:50.170438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:50.835434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:50.835400 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.066031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.065992 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.219036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.217744 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:32:51.219568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.219343 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" containerID="cri-o://0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735" gracePeriod=30 Nov 03 20:32:51.251880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.251843 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.423760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.422557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.750890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:51.750475 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.452645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:52.452610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.545311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:52.545264 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.712359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:52.708899 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.744200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:52.728651 5954 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-67mzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:52.744200 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:52.728706 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:53.718161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:53.718076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.579466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:54.579429 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:55.330574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:55.330533 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.000915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.000868 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.100684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.100649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.201667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.201631 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.557979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.557889 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.658157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.658091 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.689122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.689085 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:32:56.727192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.727142 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:32:56.788270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.788234 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:32:56.788490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.788438 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" podUID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" containerName="maas-api" containerID="cri-o://87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc" gracePeriod=30 Nov 03 20:32:56.789444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.789381 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.790732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790683 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.790877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790742 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.790877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790774 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.790877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790840 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.791066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790890 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.791066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.790933 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.791066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.791002 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.791218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.791157 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\") pod \"610daeb7-56b0-495e-945c-92b44d93709d\" (UID: \"610daeb7-56b0-495e-945c-92b44d93709d\") " Nov 03 20:32:56.791530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.791504 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.791878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.791854 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:56.792087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.792000 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.792248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.791867 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.792376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.792319 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.792376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.792339 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/610daeb7-56b0-495e-945c-92b44d93709d-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.792376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.792352 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.792568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.792479 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.795036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.794996 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.802296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.802262 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:56.802296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.802279 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "kube-api-access-jk2gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:56.802485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.802373 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token" pod "610daeb7-56b0-495e-945c-92b44d93709d" (UID: "610daeb7-56b0-495e-945c-92b44d93709d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:56.870257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.870214 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.871732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.871707 5954 generic.go:356] "Generic (PLEG): container finished" podID="610daeb7-56b0-495e-945c-92b44d93709d" containerID="0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735" exitCode=0 Nov 03 20:32:56.871893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.871774 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" Nov 03 20:32:56.871893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.871785 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerDied","Data":"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735"} Nov 03 20:32:56.871893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.871875 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv" event={"ID":"610daeb7-56b0-495e-945c-92b44d93709d","Type":"ContainerDied","Data":"d49745f50191252b15b5f891cb4d2299aab46a770d3a6f7d56bde0edc29517b1"} Nov 03 20:32:56.872072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.871907 5954 scope.go:122] "RemoveContainer" containerID="0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735" Nov 03 20:32:56.894309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894281 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.894309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894309 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk2gx\" (UniqueName: \"kubernetes.io/projected/610daeb7-56b0-495e-945c-92b44d93709d-kube-api-access-jk2gx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.894309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894322 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.894546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894331 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/610daeb7-56b0-495e-945c-92b44d93709d-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.894546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894340 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.894546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.894348 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/610daeb7-56b0-495e-945c-92b44d93709d-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.899503 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.899471 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:32:56.907232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.907198 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-67mzv"] Nov 03 20:32:56.910201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.910175 5954 scope.go:122] "RemoveContainer" containerID="3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1" Nov 03 20:32:56.924575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.924250 5954 scope.go:122] "RemoveContainer" containerID="0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735" Nov 03 20:32:56.925479 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:56.924804 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735\": container with ID starting with 0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735 not found: ID does not exist" containerID="0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735" Nov 03 20:32:56.925479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.924844 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735"} err="failed to get container status \"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735\": rpc error: code = NotFound desc = could not find container \"0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735\": container with ID starting with 0b66b3ef27caafb3784c315c5d9a98bf06b22b0349ef102475643a59cb6a6735 not found: ID does not exist" Nov 03 20:32:56.925479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.924871 5954 scope.go:122] "RemoveContainer" containerID="3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1" Nov 03 20:32:56.925858 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:56.925773 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1\": container with ID starting with 3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1 not found: ID does not exist" containerID="3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1" Nov 03 20:32:56.925858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.925805 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1"} err="failed to get container status \"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1\": rpc error: code = NotFound desc = could not find container \"3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1\": container with ID starting with 3947f662ea266d817068b9a8dc08760254d624a942955ba50f5a92de09f011e1 not found: ID does not exist" Nov 03 20:32:56.972059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:56.972024 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.388704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.388669 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.407175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.407099 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610daeb7-56b0-495e-945c-92b44d93709d" path="/var/lib/kubelet/pods/610daeb7-56b0-495e-945c-92b44d93709d/volumes" Nov 03 20:32:57.408477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.407934 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:32:57.413777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.413752 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\") pod \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " Nov 03 20:32:57.413918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.413807 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\") pod \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " Nov 03 20:32:57.413918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.413828 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\" (UniqueName: \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\") pod \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " Nov 03 20:32:57.414154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.414133 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca" pod "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" (UID: "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:57.414661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.414626 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.416913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.416889 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls" pod "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" (UID: "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:57.417289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.417261 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4" pod "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" (UID: "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f"). InnerVolumeSpecName "kube-api-access-jf6c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:57.496178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.496135 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.515706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.515670 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\") pod \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\" (UID: \"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f\") " Nov 03 20:32:57.516450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.516427 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.516565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.516454 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6c4\" (UniqueName: \"kubernetes.io/projected/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-kube-api-access-jf6c4\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.517793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.517766 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs" pod "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" (UID: "fd12b01c-dfa7-4a58-8f5d-1a75283dca1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:57.621043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.620249 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.882097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.882057 5954 generic.go:356] "Generic (PLEG): container finished" podID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" containerID="87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc" exitCode=0 Nov 03 20:32:57.882097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.882123 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" event={"ID":"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f","Type":"ContainerDied","Data":"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc"} Nov 03 20:32:57.882345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.882138 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" Nov 03 20:32:57.882345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.882177 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw" event={"ID":"fd12b01c-dfa7-4a58-8f5d-1a75283dca1f","Type":"ContainerDied","Data":"ef5647f85e3de0e9cbb41928bce7dfca001a59d6ffc7635b9127fe6c632fa9ca"} Nov 03 20:32:57.882345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.882206 5954 scope.go:122] "RemoveContainer" containerID="87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc" Nov 03 20:32:57.893779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.893759 5954 scope.go:122] "RemoveContainer" containerID="87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc" Nov 03 20:32:57.894083 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:57.894061 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc\": container with ID starting with 87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc not found: ID does not exist" containerID="87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc" Nov 03 20:32:57.894182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.894093 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc"} err="failed to get container status \"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc\": rpc error: code = NotFound desc = could not find container \"87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc\": container with ID starting with 87fe1b02ff53ff42171dbf22d4208a07374c11633deadfff4fee8d6374525ffc not found: ID does not exist" Nov 03 20:32:57.907117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.907083 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:32:57.911547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.911520 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7ae0f1-68679fb944-wqzkw"] Nov 03 20:32:57.932410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:57.932376 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.219995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.219907 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:32:58.220217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.220135 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" containerID="cri-o://7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3" gracePeriod=30 Nov 03 20:32:58.269998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.269969 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.533392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.524361 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.690164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.688432 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.799910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.799729 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.867939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.867907 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:32:58.899193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.899157 5954 generic.go:356] "Generic (PLEG): container finished" podID="16787376-4114-4876-8e0a-cd916a455f71" containerID="7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3" exitCode=0 Nov 03 20:32:58.899401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.899223 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerDied","Data":"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3"} Nov 03 20:32:58.899401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.899276 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" event={"ID":"16787376-4114-4876-8e0a-cd916a455f71","Type":"ContainerDied","Data":"0fade55172d8a73a385011698e4c8f2f2be12933a11d7db9008aa096255cd9c1"} Nov 03 20:32:58.899401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.899310 5954 scope.go:122] "RemoveContainer" containerID="7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3" Nov 03 20:32:58.901004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.900980 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v" Nov 03 20:32:58.921749 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.921719 5954 scope.go:122] "RemoveContainer" containerID="ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298" Nov 03 20:32:58.932306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.932280 5954 scope.go:122] "RemoveContainer" containerID="7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3" Nov 03 20:32:58.932651 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:58.932629 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3\": container with ID starting with 7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3 not found: ID does not exist" containerID="7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3" Nov 03 20:32:58.932755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.932659 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3"} err="failed to get container status \"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3\": rpc error: code = NotFound desc = could not find container \"7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3\": container with ID starting with 7f54be1bdb138ea5c9944a83a69ef947354f628427a6f60b58985c93047615d3 not found: ID does not exist" Nov 03 20:32:58.932755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.932679 5954 scope.go:122] "RemoveContainer" containerID="ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298" Nov 03 20:32:58.932979 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:32:58.932956 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298\": container with ID starting with ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298 not found: ID does not exist" containerID="ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298" Nov 03 20:32:58.933055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.932984 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298"} err="failed to get container status \"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298\": rpc error: code = NotFound desc = could not find container \"ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298\": container with ID starting with ba853303c46650eef4840fc40732cf1d8452c8b6827cbef1088fe2bbfc76f298 not found: ID does not exist" Nov 03 20:32:58.975190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.975153 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\") pod \"16787376-4114-4876-8e0a-cd916a455f71\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " Nov 03 20:32:58.975368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.975291 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\") pod \"16787376-4114-4876-8e0a-cd916a455f71\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " Nov 03 20:32:58.975462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.975400 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\") pod \"16787376-4114-4876-8e0a-cd916a455f71\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " Nov 03 20:32:58.975462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.975439 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\" (UniqueName: \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\") pod \"16787376-4114-4876-8e0a-cd916a455f71\" (UID: \"16787376-4114-4876-8e0a-cd916a455f71\") " Nov 03 20:32:58.975568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.975547 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca" pod "16787376-4114-4876-8e0a-cd916a455f71" (UID: "16787376-4114-4876-8e0a-cd916a455f71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:58.976212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.976190 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16787376-4114-4876-8e0a-cd916a455f71-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:58.977524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.977496 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs" pod "16787376-4114-4876-8e0a-cd916a455f71" (UID: "16787376-4114-4876-8e0a-cd916a455f71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.977524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.977518 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls" pod "16787376-4114-4876-8e0a-cd916a455f71" (UID: "16787376-4114-4876-8e0a-cd916a455f71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:58.977709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:58.977593 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k" pod "16787376-4114-4876-8e0a-cd916a455f71" (UID: "16787376-4114-4876-8e0a-cd916a455f71"). InnerVolumeSpecName "kube-api-access-qg46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:59.077936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.077855 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.077936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.077886 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16787376-4114-4876-8e0a-cd916a455f71-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.077936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.077896 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg46k\" (UniqueName: \"kubernetes.io/projected/16787376-4114-4876-8e0a-cd916a455f71-kube-api-access-qg46k\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:59.129254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.129215 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:59.227681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.227609 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:32:59.231962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.231936 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6cf8ae-568b98fc7f-j4q2v"] Nov 03 20:32:59.408393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.408360 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16787376-4114-4876-8e0a-cd916a455f71" path="/var/lib/kubelet/pods/16787376-4114-4876-8e0a-cd916a455f71/volumes" Nov 03 20:32:59.408903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.408887 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" path="/var/lib/kubelet/pods/fd12b01c-dfa7-4a58-8f5d-1a75283dca1f/volumes" Nov 03 20:32:59.592863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:32:59.592811 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:00.260316 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:00.260267 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:33:00.293368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:00.293333 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:01.126133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:01.122658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:01.986618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:01.986582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.207359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:02.207318 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:33:02.221143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:02.219394 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:33:02.290181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:02.287487 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.386804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:02.386767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.005679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:03.005648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.883561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:03.883528 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.486018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:04.485983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.678135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:04.678015 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.845184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:04.834582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:04.933523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:04.933491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.575644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.575611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.678945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.678905 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.909673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.909628 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.915541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.915498 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:33:05.915838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.915801 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" containerID="cri-o://e0f289509ba14dac3dbe99ab2c6101345fd7376e01d619ae33a88b2e58a92197" gracePeriod=30 Nov 03 20:33:05.918971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.918941 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:33:05.919300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:05.919246 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" containerID="cri-o://8d43f9f53a7d3cfecf0b11e8ced37b98e5aa577e9aa282078d1743c6a560e7bf" gracePeriod=30 Nov 03 20:33:06.045396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.028564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:06.302995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.302903 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:06.419183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.413913 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:06.635356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.635323 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:33:06.635816 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.635586 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" containerID="cri-o://ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e" gracePeriod=30 Nov 03 20:33:06.682319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:06.682263 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:07.008807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.008715 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:07.207654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.207614 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" podUID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" containerName="payload-processing" containerID="cri-o://1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1" gracePeriod=25 Nov 03 20:33:07.220177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.220137 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" containerName="payload-pre-processing" containerID="cri-o://f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5" gracePeriod=25 Nov 03 20:33:07.516370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.516344 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:33:07.579742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.579707 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:33:07.598739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.598706 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\") pod \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " Nov 03 20:33:07.598946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.598781 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\" (UniqueName: \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\") pod \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\" (UID: \"0d5b16f2-bfeb-4339-9660-32090c2e89ab\") " Nov 03 20:33:07.599175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.599142 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume" pod "0d5b16f2-bfeb-4339-9660-32090c2e89ab" (UID: "0d5b16f2-bfeb-4339-9660-32090c2e89ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.599577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.599557 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5b16f2-bfeb-4339-9660-32090c2e89ab-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.601778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.601749 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs" pod "0d5b16f2-bfeb-4339-9660-32090c2e89ab" (UID: "0d5b16f2-bfeb-4339-9660-32090c2e89ab"). InnerVolumeSpecName "kube-api-access-67pqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.700975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.700942 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\" (UniqueName: \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\") pod \"9de87552-80de-44bc-9487-058fcd3cbabe\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " Nov 03 20:33:07.701414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.700990 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\") pod \"9de87552-80de-44bc-9487-058fcd3cbabe\" (UID: \"9de87552-80de-44bc-9487-058fcd3cbabe\") " Nov 03 20:33:07.701499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.701407 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume" pod "9de87552-80de-44bc-9487-058fcd3cbabe" (UID: "9de87552-80de-44bc-9487-058fcd3cbabe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:07.701679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.701660 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pqs\" (UniqueName: \"kubernetes.io/projected/0d5b16f2-bfeb-4339-9660-32090c2e89ab-kube-api-access-67pqs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.701733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.701683 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de87552-80de-44bc-9487-058fcd3cbabe-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.703441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.703419 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6" pod "9de87552-80de-44bc-9487-058fcd3cbabe" (UID: "9de87552-80de-44bc-9487-058fcd3cbabe"). InnerVolumeSpecName "kube-api-access-pthz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.802821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:07.802731 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthz6\" (UniqueName: \"kubernetes.io/projected/9de87552-80de-44bc-9487-058fcd3cbabe-kube-api-access-pthz6\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:08.078184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.078068 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.201313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.201278 5954 generic.go:356] "Generic (PLEG): container finished" podID="9de87552-80de-44bc-9487-058fcd3cbabe" containerID="f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5" exitCode=0 Nov 03 20:33:08.201520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.201352 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" Nov 03 20:33:08.201520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.201378 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" event={"ID":"9de87552-80de-44bc-9487-058fcd3cbabe","Type":"ContainerDied","Data":"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5"} Nov 03 20:33:08.201520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.201442 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb" event={"ID":"9de87552-80de-44bc-9487-058fcd3cbabe","Type":"ContainerDied","Data":"f557f52600cce58d605732f720a833a34f6199d0e2f490306b0a9292a6207b6f"} Nov 03 20:33:08.201520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.201478 5954 scope.go:122] "RemoveContainer" containerID="f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5" Nov 03 20:33:08.202985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.202963 5954 generic.go:356] "Generic (PLEG): container finished" podID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" containerID="1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1" exitCode=0 Nov 03 20:33:08.203100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.203007 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" Nov 03 20:33:08.203100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.203061 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" event={"ID":"0d5b16f2-bfeb-4339-9660-32090c2e89ab","Type":"ContainerDied","Data":"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1"} Nov 03 20:33:08.203248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.203144 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w" event={"ID":"0d5b16f2-bfeb-4339-9660-32090c2e89ab","Type":"ContainerDied","Data":"195d4d3c2d7f395fd05e766d961531e61fd94af94d2f7a4cbba10192ed16b92e"} Nov 03 20:33:08.212546 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.212344 5954 scope.go:122] "RemoveContainer" containerID="f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5" Nov 03 20:33:08.212658 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:08.212634 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5\": container with ID starting with f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5 not found: ID does not exist" containerID="f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5" Nov 03 20:33:08.212724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.212668 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5"} err="failed to get container status \"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5\": rpc error: code = NotFound desc = could not find container \"f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5\": container with ID starting with f6e0e99f0df6e7588bdab7621bf77a0859389fbf0e61a9941430675906a0d4c5 not found: ID does not exist" Nov 03 20:33:08.212724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.212686 5954 scope.go:122] "RemoveContainer" containerID="1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1" Nov 03 20:33:08.236272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.236248 5954 scope.go:122] "RemoveContainer" containerID="1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1" Nov 03 20:33:08.236633 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:08.236611 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1\": container with ID starting with 1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1 not found: ID does not exist" containerID="1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1" Nov 03 20:33:08.236692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.236643 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1"} err="failed to get container status \"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1\": rpc error: code = NotFound desc = could not find container \"1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1\": container with ID starting with 1e5d2904d01dcd2dc7b381c6543f7866b5555f71fcd0722c578f8c5959edf6f1 not found: ID does not exist" Nov 03 20:33:08.246050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.246025 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:33:08.249880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.249858 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-90ef9c-6dc7949ff4-kgm2w"] Nov 03 20:33:08.262987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.262958 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:33:08.266467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.266442 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-90ef9c-6f949db6f9-hslsb"] Nov 03 20:33:08.972631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.972594 5954 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:08.973047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:08.972652 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:09.142190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:09.140988 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:09.416280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:09.416247 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" path="/var/lib/kubelet/pods/0d5b16f2-bfeb-4339-9660-32090c2e89ab/volumes" Nov 03 20:33:09.416702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:09.416686 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" path="/var/lib/kubelet/pods/9de87552-80de-44bc-9487-058fcd3cbabe/volumes" Nov 03 20:33:09.904180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:09.904146 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.604190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:10.604156 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:11.231007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.230983 5954 generic.go:356] "Generic (PLEG): container finished" podID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerID="e0f289509ba14dac3dbe99ab2c6101345fd7376e01d619ae33a88b2e58a92197" exitCode=0 Nov 03 20:33:11.231087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.231050 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerDied","Data":"e0f289509ba14dac3dbe99ab2c6101345fd7376e01d619ae33a88b2e58a92197"} Nov 03 20:33:11.231171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.231150 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" event={"ID":"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd","Type":"ContainerDied","Data":"8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2"} Nov 03 20:33:11.231227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.231190 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8674c69afb6921f02cb424f3c63d180ee567399d77e9bddd1df99c8930eb4bf2" Nov 03 20:33:11.231227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.231210 5954 scope.go:122] "RemoveContainer" containerID="5eeae0ef3b0b7fb0c72daef973c5db9234b8fd61096a76ecf042bc2f06c05a33" Nov 03 20:33:11.232617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.232596 5954 generic.go:356] "Generic (PLEG): container finished" podID="175752d9-a28f-4352-9e81-5dc71de5df45" containerID="8d43f9f53a7d3cfecf0b11e8ced37b98e5aa577e9aa282078d1743c6a560e7bf" exitCode=0 Nov 03 20:33:11.232714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.232674 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" event={"ID":"175752d9-a28f-4352-9e81-5dc71de5df45","Type":"ContainerDied","Data":"8d43f9f53a7d3cfecf0b11e8ced37b98e5aa577e9aa282078d1743c6a560e7bf"} Nov 03 20:33:11.255543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.255517 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:33:11.258194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.258174 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:33:11.330060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.330025 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401146 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401221 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401293 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401361 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401432 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401467 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401548 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401607 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401656 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401788 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401866 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401944 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.401986 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.402065 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.402155 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.402182 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.402227 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\") pod \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\" (UID: \"99d7cb77-77fe-4bf2-bef6-b0374b2f17cd\") " Nov 03 20:33:11.406158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.402287 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\") pod \"175752d9-a28f-4352-9e81-5dc71de5df45\" (UID: \"175752d9-a28f-4352-9e81-5dc71de5df45\") " Nov 03 20:33:11.422263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.422191 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.422585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.422418 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.422716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.422592 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.424748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.424475 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.425748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.424913 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.427161 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.427553 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.427570 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.428567 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.428852 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.429177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.429040 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.429931 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.430052 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "kube-api-access-n9tzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.430910 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.434155 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "kube-api-access-d6zz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.434149 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy" pod "175752d9-a28f-4352-9e81-5dc71de5df45" (UID: "175752d9-a28f-4352-9e81-5dc71de5df45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.437179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.436497 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.438148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.438089 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo" pod "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" (UID: "99d7cb77-77fe-4bf2-bef6-b0374b2f17cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:11.522850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522809 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175752d9-a28f-4352-9e81-5dc71de5df45-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.522850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522844 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zz5\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-kube-api-access-d6zz5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522861 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9tzg\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-kube-api-access-n9tzg\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522874 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522883 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522895 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522908 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522917 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522926 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175752d9-a28f-4352-9e81-5dc71de5df45-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522938 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522952 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522963 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175752d9-a28f-4352-9e81-5dc71de5df45-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522971 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522981 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522991 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.522999 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175752d9-a28f-4352-9e81-5dc71de5df45-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.523007 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.523154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.523017 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.762680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.762641 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" containerID="cri-o://6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549" gracePeriod=45 Nov 03 20:33:11.910207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.909881 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:33:11.928528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928496 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928574 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928630 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928667 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928704 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928777 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.928872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928814 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.928872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928824 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.929094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.928914 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.929094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929004 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.929094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929025 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.929094 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929062 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\") pod \"a2f6a4cc-ae68-474e-81ba-7e1675060159\" (UID: \"a2f6a4cc-ae68-474e-81ba-7e1675060159\") " Nov 03 20:33:11.929281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929150 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.929510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929489 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:11.929835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929815 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.929925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929841 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2f6a4cc-ae68-474e-81ba-7e1675060159-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.929925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929858 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.929925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929872 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.929925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.929888 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.931096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.931070 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.931312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.931285 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:11.931415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.931391 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "kube-api-access-c6rs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.931415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:11.931406 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy" pod "a2f6a4cc-ae68-474e-81ba-7e1675060159" (UID: "a2f6a4cc-ae68-474e-81ba-7e1675060159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.031732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.031639 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.031732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.031680 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.031732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.031697 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6rs8\" (UniqueName: \"kubernetes.io/projected/a2f6a4cc-ae68-474e-81ba-7e1675060159-kube-api-access-c6rs8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.031732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.031712 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2f6a4cc-ae68-474e-81ba-7e1675060159-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.068903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.068875 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:33:12.132719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.132679 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.132939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.132762 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\" (UniqueName: \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.132939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.132874 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.133079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.132975 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.133079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.132986 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.133079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133020 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\" (UniqueName: \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.133298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133096 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\") pod \"31bec522-d1ff-47fc-b8df-b862eb1faae5\" (UID: \"31bec522-d1ff-47fc-b8df-b862eb1faae5\") " Nov 03 20:33:12.133298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133189 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.133704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133675 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.133876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133854 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.133994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133880 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.133994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.133894 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.135565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.135527 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.135995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.135968 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "kube-api-access-nx767". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.136399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.136375 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm" pod "31bec522-d1ff-47fc-b8df-b862eb1faae5" (UID: "31bec522-d1ff-47fc-b8df-b862eb1faae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.196500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.196461 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.235972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.235942 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx767\" (UniqueName: \"kubernetes.io/projected/31bec522-d1ff-47fc-b8df-b862eb1faae5-kube-api-access-nx767\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.235972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.235969 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bec522-d1ff-47fc-b8df-b862eb1faae5-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.235972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.235981 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bec522-d1ff-47fc-b8df-b862eb1faae5-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.241987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.241952 5954 generic.go:356] "Generic (PLEG): container finished" podID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerID="6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549" exitCode=0 Nov 03 20:33:12.242153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.241996 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" Nov 03 20:33:12.242153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.242032 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" event={"ID":"31bec522-d1ff-47fc-b8df-b862eb1faae5","Type":"ContainerDied","Data":"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549"} Nov 03 20:33:12.242153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.242073 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh" event={"ID":"31bec522-d1ff-47fc-b8df-b862eb1faae5","Type":"ContainerDied","Data":"3b2bbd3e8be1ff3aa5aeb85447d2eec9a1e63d94d40097336009718d7392bbd4"} Nov 03 20:33:12.242153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.242090 5954 scope.go:122] "RemoveContainer" containerID="6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549" Nov 03 20:33:12.244146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.244082 5954 generic.go:356] "Generic (PLEG): container finished" podID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerID="ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e" exitCode=0 Nov 03 20:33:12.244146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.244127 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerDied","Data":"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e"} Nov 03 20:33:12.244297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.244176 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" Nov 03 20:33:12.244297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.244182 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk" event={"ID":"a2f6a4cc-ae68-474e-81ba-7e1675060159","Type":"ContainerDied","Data":"f2ccb153196eaebd8b412663369cc3072dcf3dd85e33735f6f48cc10c402b445"} Nov 03 20:33:12.245857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.245816 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h" Nov 03 20:33:12.248193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.248175 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" Nov 03 20:33:12.248193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.248185 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx" event={"ID":"175752d9-a28f-4352-9e81-5dc71de5df45","Type":"ContainerDied","Data":"f51106f8aff0f9803901342de6aa091434af35bae4888143a86e81b2055df4e1"} Nov 03 20:33:12.261075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.261051 5954 scope.go:122] "RemoveContainer" containerID="6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549" Nov 03 20:33:12.261444 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:12.261424 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549\": container with ID starting with 6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549 not found: ID does not exist" containerID="6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549" Nov 03 20:33:12.261517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.261454 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549"} err="failed to get container status \"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549\": rpc error: code = NotFound desc = could not find container \"6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549\": container with ID starting with 6018183ad69fc2e48fb50ea0cb580ecdc468aa10099712779998fc5ff2ca4549 not found: ID does not exist" Nov 03 20:33:12.261517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.261474 5954 scope.go:122] "RemoveContainer" containerID="ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e" Nov 03 20:33:12.272517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.272498 5954 scope.go:122] "RemoveContainer" containerID="f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217" Nov 03 20:33:12.310689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.310666 5954 scope.go:122] "RemoveContainer" containerID="ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e" Nov 03 20:33:12.310783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.310770 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:33:12.311059 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:12.311038 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e\": container with ID starting with ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e not found: ID does not exist" containerID="ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e" Nov 03 20:33:12.311232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.311067 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e"} err="failed to get container status \"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e\": rpc error: code = NotFound desc = could not find container \"ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e\": container with ID starting with ff5bcc6171f34755c0a187a1515bcdefd10cbd2c3bec9a9878455c620b6a491e not found: ID does not exist" Nov 03 20:33:12.311232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.311088 5954 scope.go:122] "RemoveContainer" containerID="f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217" Nov 03 20:33:12.311535 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:12.311479 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217\": container with ID starting with f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217 not found: ID does not exist" containerID="f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217" Nov 03 20:33:12.311535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.311512 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217"} err="failed to get container status \"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217\": rpc error: code = NotFound desc = could not find container \"f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217\": container with ID starting with f2b9247a664ee05192a9afdc26aa714e06ce9b61f4d589147e4e5882f0e6d217 not found: ID does not exist" Nov 03 20:33:12.311535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.311531 5954 scope.go:122] "RemoveContainer" containerID="8d43f9f53a7d3cfecf0b11e8ced37b98e5aa577e9aa282078d1743c6a560e7bf" Nov 03 20:33:12.335127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.326485 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-z5ghk"] Nov 03 20:33:12.344511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.339408 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:33:12.346352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.346321 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-s4p8h"] Nov 03 20:33:12.353371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.353310 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:33:12.359906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.359872 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-vhkxx"] Nov 03 20:33:12.403351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.403314 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:33:12.408404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:12.408368 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-w2-90ef9c-kserve-868d7f6d77-f4xqh"] Nov 03 20:33:13.217913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:13.217867 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.408444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:13.408407 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" path="/var/lib/kubelet/pods/175752d9-a28f-4352-9e81-5dc71de5df45/volumes" Nov 03 20:33:13.408929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:13.408915 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" path="/var/lib/kubelet/pods/31bec522-d1ff-47fc-b8df-b862eb1faae5/volumes" Nov 03 20:33:13.409386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:13.409372 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" path="/var/lib/kubelet/pods/99d7cb77-77fe-4bf2-bef6-b0374b2f17cd/volumes" Nov 03 20:33:13.410226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:13.410209 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" path="/var/lib/kubelet/pods/a2f6a4cc-ae68-474e-81ba-7e1675060159/volumes" Nov 03 20:33:14.102247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:14.102215 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.778236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:14.778196 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:15.499584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:15.499551 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:15.603527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:15.603491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.379438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:16.379353 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.688168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:16.688063 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.791837 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:16.791801 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:17.289147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:17.289072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:17.443451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:17.443380 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:17.549035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:17.548946 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:18.286767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:18.286731 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.080132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:19.074901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.809368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:19.809320 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.328166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.328125 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:33:20.328731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.328385 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" podUID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" containerName="maas-api" containerID="cri-o://f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a" gracePeriod=30 Nov 03 20:33:20.392499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.392449 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.772348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.772200 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.883340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.882995 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.912466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.912439 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:33:20.986238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.986192 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\") pod \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " Nov 03 20:33:20.986446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.986318 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\") pod \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " Nov 03 20:33:20.986446 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.986345 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\" (UniqueName: \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\") pod \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " Nov 03 20:33:20.986570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.986450 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\") pod \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\" (UID: \"8cacbf0a-b64e-4c51-a22b-742e2f2e6924\") " Nov 03 20:33:20.986913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.986882 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca" pod "8cacbf0a-b64e-4c51-a22b-742e2f2e6924" (UID: "8cacbf0a-b64e-4c51-a22b-742e2f2e6924"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:20.987277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.987249 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:20.988565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.988533 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls" pod "8cacbf0a-b64e-4c51-a22b-742e2f2e6924" (UID: "8cacbf0a-b64e-4c51-a22b-742e2f2e6924"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:20.988674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.988585 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs" pod "8cacbf0a-b64e-4c51-a22b-742e2f2e6924" (UID: "8cacbf0a-b64e-4c51-a22b-742e2f2e6924"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:20.988741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:20.988670 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz" pod "8cacbf0a-b64e-4c51-a22b-742e2f2e6924" (UID: "8cacbf0a-b64e-4c51-a22b-742e2f2e6924"). InnerVolumeSpecName "kube-api-access-z2cjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:21.050139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.050034 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:21.088865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.088823 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.088865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.088865 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.089079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.088881 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2cjz\" (UniqueName: \"kubernetes.io/projected/8cacbf0a-b64e-4c51-a22b-742e2f2e6924-kube-api-access-z2cjz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:21.425379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.425339 5954 generic.go:356] "Generic (PLEG): container finished" podID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" containerID="f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a" exitCode=0 Nov 03 20:33:21.425852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.425389 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" Nov 03 20:33:21.425852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.425412 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" event={"ID":"8cacbf0a-b64e-4c51-a22b-742e2f2e6924","Type":"ContainerDied","Data":"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a"} Nov 03 20:33:21.425852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.425482 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk" event={"ID":"8cacbf0a-b64e-4c51-a22b-742e2f2e6924","Type":"ContainerDied","Data":"d377169263a72115c8e8d1b1d04d982c02655b620a22ed22746c91c0a930f231"} Nov 03 20:33:21.425852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.425513 5954 scope.go:122] "RemoveContainer" containerID="f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a" Nov 03 20:33:21.437156 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.437087 5954 scope.go:122] "RemoveContainer" containerID="f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a" Nov 03 20:33:21.437499 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:21.437478 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a\": container with ID starting with f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a not found: ID does not exist" containerID="f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a" Nov 03 20:33:21.437600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.437509 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a"} err="failed to get container status \"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a\": rpc error: code = NotFound desc = could not find container \"f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a\": container with ID starting with f829bb7bf555f80e7a05d90a858772495fba8cef9f5f5d0079407bce1444c58a not found: ID does not exist" Nov 03 20:33:21.444603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.444530 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:33:21.449259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.449235 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d0898b-77b4889958-cjfgk"] Nov 03 20:33:21.690394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:21.690294 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.358036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.358002 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.467661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.465495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.574584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.574540 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.677614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.677525 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:33:22.677835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.677808 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" containerID="cri-o://72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007" gracePeriod=30 Nov 03 20:33:22.720771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:22.720740 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:23.405358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:23.405328 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" path="/var/lib/kubelet/pods/8cacbf0a-b64e-4c51-a22b-742e2f2e6924/volumes" Nov 03 20:33:23.584786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:23.584745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:24.376002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:24.375967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.288525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:25.288489 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.387181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:25.387089 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.570417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:25.570331 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:33:26.101130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:26.099691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:27.014783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:27.014645 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:27.967999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:27.967973 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:33:27.975191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:27.975161 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:28.048648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048611 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048657 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048735 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048774 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048807 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048907 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.048956 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.049060 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.049291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.049117 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\") pod \"6761899d-1c05-4ecb-8373-27c9c5813ac0\" (UID: \"6761899d-1c05-4ecb-8373-27c9c5813ac0\") " Nov 03 20:33:28.072578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.072528 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.074993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.074960 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.086951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.080657 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.086951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.081251 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:28.088326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.088284 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.090187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.090158 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:28.097784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.096091 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:28.097784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.097455 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "kube-api-access-54lnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.098236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.098084 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token" pod "6761899d-1c05-4ecb-8373-27c9c5813ac0" (UID: "6761899d-1c05-4ecb-8373-27c9c5813ac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:28.167728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167685 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.167728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167726 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167744 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lnt\" (UniqueName: \"kubernetes.io/projected/6761899d-1c05-4ecb-8373-27c9c5813ac0-kube-api-access-54lnt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167758 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167771 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167783 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167795 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167808 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6761899d-1c05-4ecb-8373-27c9c5813ac0-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.168020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.167819 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6761899d-1c05-4ecb-8373-27c9c5813ac0-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:28.508686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.508652 5954 generic.go:356] "Generic (PLEG): container finished" podID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerID="72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007" exitCode=0 Nov 03 20:33:28.508917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.508734 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" Nov 03 20:33:28.508917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.508746 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerDied","Data":"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007"} Nov 03 20:33:28.508917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.508801 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg" event={"ID":"6761899d-1c05-4ecb-8373-27c9c5813ac0","Type":"ContainerDied","Data":"760c3fba06abbf6e85f8c34c018c2eadf957f849aefa032b9dc08139ce240996"} Nov 03 20:33:28.508917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.508831 5954 scope.go:122] "RemoveContainer" containerID="72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007" Nov 03 20:33:28.519863 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.519845 5954 scope.go:122] "RemoveContainer" containerID="de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f" Nov 03 20:33:28.530044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.530027 5954 scope.go:122] "RemoveContainer" containerID="72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007" Nov 03 20:33:28.530357 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:28.530332 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007\": container with ID starting with 72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007 not found: ID does not exist" containerID="72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007" Nov 03 20:33:28.530425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.530396 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007"} err="failed to get container status \"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007\": rpc error: code = NotFound desc = could not find container \"72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007\": container with ID starting with 72b28205708ea8126634c834eaaf1a34b6bdb6a33b0602162bc1a4ca222bc007 not found: ID does not exist" Nov 03 20:33:28.530494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.530426 5954 scope.go:122] "RemoveContainer" containerID="de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f" Nov 03 20:33:28.531692 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:28.530804 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f\": container with ID starting with de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f not found: ID does not exist" containerID="de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f" Nov 03 20:33:28.531692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.530845 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f"} err="failed to get container status \"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f\": rpc error: code = NotFound desc = could not find container \"de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f\": container with ID starting with de4ffbac2a081584eba8193cba6bf2068670e77acc298c87b522cb6425d2cf3f not found: ID does not exist" Nov 03 20:33:28.534647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.534625 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:33:28.539520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.539008 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c684d-openshift-default-7dcb6dd65c-q6vgg"] Nov 03 20:33:28.685146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.685098 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:28.888166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.888134 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:28.990604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:28.990568 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:29.213325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:29.213228 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:33:29.213803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:29.213511 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" containerID="cri-o://301e46984ef0c7e56ec73043ba458c9f34d8f6367b807d426f1c8ae6d7a2800b" gracePeriod=30 Nov 03 20:33:29.271472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:29.271426 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:29.476188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:29.470668 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" path="/var/lib/kubelet/pods/6761899d-1c05-4ecb-8373-27c9c5813ac0/volumes" Nov 03 20:33:29.621261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:29.621222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:30.258963 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:30.258928 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:33:30.489786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:30.489756 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:31.429145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:31.429079 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.070260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.070226 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:33:32.070482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.070437 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" podUID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" containerName="maas-api" containerID="cri-o://2b1791a34d0121b3bbe1bda9896817f8b9992a069c30e8986633f2930b38dad1" gracePeriod=30 Nov 03 20:33:32.119474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.119438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.365128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.364467 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.546563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.546532 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" containerID="2b1791a34d0121b3bbe1bda9896817f8b9992a069c30e8986633f2930b38dad1" exitCode=0 Nov 03 20:33:32.546938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.546607 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" event={"ID":"ff143bfd-2b06-48c0-8a32-0062ce724b4d","Type":"ContainerDied","Data":"2b1791a34d0121b3bbe1bda9896817f8b9992a069c30e8986633f2930b38dad1"} Nov 03 20:33:32.573844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.573813 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:33:32.653331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.653234 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.710176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710144 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\") pod \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " Nov 03 20:33:32.710379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710257 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\" (UniqueName: \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\") pod \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " Nov 03 20:33:32.710379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710290 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\") pod \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " Nov 03 20:33:32.710379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710337 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\") pod \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\" (UID: \"ff143bfd-2b06-48c0-8a32-0062ce724b4d\") " Nov 03 20:33:32.710570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710481 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca" pod "ff143bfd-2b06-48c0-8a32-0062ce724b4d" (UID: "ff143bfd-2b06-48c0-8a32-0062ce724b4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:32.710911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.710889 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff143bfd-2b06-48c0-8a32-0062ce724b4d-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.712451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.712421 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr" pod "ff143bfd-2b06-48c0-8a32-0062ce724b4d" (UID: "ff143bfd-2b06-48c0-8a32-0062ce724b4d"). InnerVolumeSpecName "kube-api-access-9gvmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:32.712547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.712471 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs" pod "ff143bfd-2b06-48c0-8a32-0062ce724b4d" (UID: "ff143bfd-2b06-48c0-8a32-0062ce724b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:32.712547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.712492 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls" pod "ff143bfd-2b06-48c0-8a32-0062ce724b4d" (UID: "ff143bfd-2b06-48c0-8a32-0062ce724b4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:32.776711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.776670 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.812217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.812183 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvmr\" (UniqueName: \"kubernetes.io/projected/ff143bfd-2b06-48c0-8a32-0062ce724b4d-kube-api-access-9gvmr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.812217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.812216 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.812431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:32.812228 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff143bfd-2b06-48c0-8a32-0062ce724b4d-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:33.168617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.168585 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.408392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.408346 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.516060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.516008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.560936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.560899 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" event={"ID":"ff143bfd-2b06-48c0-8a32-0062ce724b4d","Type":"ContainerDied","Data":"1ea774342f2026489b84e239896d4a9d38e8e1f166222d5c0325ed5d67ded32a"} Nov 03 20:33:33.561466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.560962 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n" Nov 03 20:33:33.562208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.560971 5954 scope.go:122] "RemoveContainer" containerID="2b1791a34d0121b3bbe1bda9896817f8b9992a069c30e8986633f2930b38dad1" Nov 03 20:33:33.592757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.592727 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:33:33.597405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.597375 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ca474-74549bff6f-twt2n"] Nov 03 20:33:33.615570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.615500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.813204 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.812930 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.955136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:33.954292 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.057358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.057308 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.156702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.156665 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.401959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.401662 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.572277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.572244 5954 generic.go:356] "Generic (PLEG): container finished" podID="7188424c-b153-4a5b-86b5-074c5560eba9" containerID="301e46984ef0c7e56ec73043ba458c9f34d8f6367b807d426f1c8ae6d7a2800b" exitCode=0 Nov 03 20:33:34.572677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.572379 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerDied","Data":"301e46984ef0c7e56ec73043ba458c9f34d8f6367b807d426f1c8ae6d7a2800b"} Nov 03 20:33:34.572677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.572459 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" event={"ID":"7188424c-b153-4a5b-86b5-074c5560eba9","Type":"ContainerDied","Data":"b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3"} Nov 03 20:33:34.572677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.572489 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12c24b209514620172f04eaab33108111f7699729d3bbefc33074b3ff09fdd3" Nov 03 20:33:34.572677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.572509 5954 scope.go:122] "RemoveContainer" containerID="f79b3ee784401b302ef453a13d02ee76ee0cb532df6dde2c4be2e291e3ce2843" Nov 03 20:33:34.576943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.576922 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:33:34.620944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.620910 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.650737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.650704 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.650929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.650783 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.650959 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651015 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651073 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651127 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651125 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.651328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651225 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651268 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651346 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.651444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651372 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\") pod \"7188424c-b153-4a5b-86b5-074c5560eba9\" (UID: \"7188424c-b153-4a5b-86b5-074c5560eba9\") " Nov 03 20:33:34.651929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651888 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.651929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651894 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.652098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.651941 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:34.652098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.652058 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.652098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.652079 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.652098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.652093 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.652384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.652122 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.653676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.653650 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "kube-api-access-297n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:34.653867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.653841 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:34.653929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.653892 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:34.654413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.654385 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy" pod "7188424c-b153-4a5b-86b5-074c5560eba9" (UID: "7188424c-b153-4a5b-86b5-074c5560eba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:34.753561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.753529 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.753561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.753557 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7188424c-b153-4a5b-86b5-074c5560eba9-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.753561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.753567 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7188424c-b153-4a5b-86b5-074c5560eba9-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.753815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.753576 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7188424c-b153-4a5b-86b5-074c5560eba9-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:34.753815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:34.753586 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297n8\" (UniqueName: \"kubernetes.io/projected/7188424c-b153-4a5b-86b5-074c5560eba9-kube-api-access-297n8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:35.404451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.404415 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:35.407022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.406993 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" path="/var/lib/kubelet/pods/ff143bfd-2b06-48c0-8a32-0062ce724b4d/volumes" Nov 03 20:33:35.507599 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.507564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:35.580035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.580009 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc" Nov 03 20:33:35.623138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.623073 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:33:35.629235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:35.629202 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d0898b-openshift-default-798d8496f5-5zclc"] Nov 03 20:33:36.199226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:36.199190 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:36.996583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:36.996548 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:37.414871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.414828 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" path="/var/lib/kubelet/pods/7188424c-b153-4a5b-86b5-074c5560eba9/volumes" Nov 03 20:33:37.612812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.612776 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:33:37.613081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.613031 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" containerName="discovery" containerID="cri-o://38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3" gracePeriod=30 Nov 03 20:33:37.633972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.633937 5954 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lth6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:8080/ready\": dial tcp 10.134.0.125:8080: connect: connection refused" start-of-body= Nov 03 20:33:37.634123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.634014 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.125:8080/ready\": dial tcp 10.134.0.125:8080: connect: connection refused" Nov 03 20:33:37.885122 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.885075 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:37.918480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:37.918454 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:33:38.011763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.010910 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.011763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.010958 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.011763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.011086 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.012378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.011998 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.012378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.012085 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.012378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.012178 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.012378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.012266 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\") pod \"78769a77-eb1b-4e93-b3ff-454954858b61\" (UID: \"78769a77-eb1b-4e93-b3ff-454954858b61\") " Nov 03 20:33:38.015183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.013838 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:38.017656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.017442 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "kube-api-access-lz96r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:38.018700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.018531 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:38.021121 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.020906 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.021250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.021229 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.022068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.021645 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:38.022951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.022509 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs" pod "78769a77-eb1b-4e93-b3ff-454954858b61" (UID: "78769a77-eb1b-4e93-b3ff-454954858b61"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:38.114659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114618 5954 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-kubeconfig\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114652 5954 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-ca-configmap\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114665 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz96r\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-kube-api-access-lz96r\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114673 5954 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78769a77-eb1b-4e93-b3ff-454954858b61-local-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114684 5954 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-cacerts\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114692 5954 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78769a77-eb1b-4e93-b3ff-454954858b61-istio-csr-dns-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.114925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.114700 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78769a77-eb1b-4e93-b3ff-454954858b61-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:38.602855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.602820 5954 generic.go:356] "Generic (PLEG): container finished" podID="78769a77-eb1b-4e93-b3ff-454954858b61" containerID="38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3" exitCode=0 Nov 03 20:33:38.603051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.602872 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" Nov 03 20:33:38.603051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.602898 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" event={"ID":"78769a77-eb1b-4e93-b3ff-454954858b61","Type":"ContainerDied","Data":"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3"} Nov 03 20:33:38.603051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.602953 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n" event={"ID":"78769a77-eb1b-4e93-b3ff-454954858b61","Type":"ContainerDied","Data":"9e0573e616767dfba2e7836d2082536a795bc055bea64f2a8a0b46598754003f"} Nov 03 20:33:38.603051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.602980 5954 scope.go:122] "RemoveContainer" containerID="38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3" Nov 03 20:33:38.606515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.606478 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:38.615983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.615844 5954 scope.go:122] "RemoveContainer" containerID="38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3" Nov 03 20:33:38.616197 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:38.616175 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3\": container with ID starting with 38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3 not found: ID does not exist" containerID="38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3" Nov 03 20:33:38.616242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.616209 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3"} err="failed to get container status \"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3\": rpc error: code = NotFound desc = could not find container \"38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3\": container with ID starting with 38d9a1001b4a717507fa3e213d9d1b4fbb2e773dfa7d0c1826a038625e2abcd3 not found: ID does not exist" Nov 03 20:33:38.628760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.628713 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:33:38.635357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:38.635326 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lth6n"] Nov 03 20:33:39.400506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:39.400454 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:39.406057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:39.406025 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" path="/var/lib/kubelet/pods/78769a77-eb1b-4e93-b3ff-454954858b61/volumes" Nov 03 20:33:39.919532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:39.919490 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.042233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.042183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.163131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.155620 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.344185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.338177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.467495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.467458 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:33:40.467997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.467727 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" containerID="cri-o://31b340494a692fca9dbceed420d9a11d79484aa1deb41e46ddb9569ef5c75caa" gracePeriod=30 Nov 03 20:33:40.493461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.493429 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:33:40.493667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.493641 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" containerID="cri-o://6d20c4f85f833655a5ac2d7ef4a034b0ec3eb9c8270b22e146fd52d4fdcd5e7b" gracePeriod=30 Nov 03 20:33:40.524336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.524283 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.910049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910012 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh"] Nov 03 20:33:40.910741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910717 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" containerName="maas-api" Nov 03 20:33:40.910741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910738 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910754 5954 cpu_manager.go:414] "containerMap: removing container" podUID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910761 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910767 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910772 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910780 5954 cpu_manager.go:414] "containerMap: removing container" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910785 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910795 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910799 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5a9d56-ffd2-4307-832a-30afacf4b48a" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910809 5954 cpu_manager.go:414] "containerMap: removing container" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910814 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910822 5954 cpu_manager.go:414] "containerMap: removing container" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" containerName="discovery" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910826 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" containerName="discovery" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910843 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910847 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910854 5954 cpu_manager.go:414] "containerMap: removing container" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910858 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910866 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910870 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910876 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910880 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" containerName="maas-api" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910887 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910892 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910905 5954 cpu_manager.go:414] "containerMap: removing container" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:40.910906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910911 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910920 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" containerName="payload-processing" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910924 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" containerName="payload-processing" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910931 5954 cpu_manager.go:414] "containerMap: removing container" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910935 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910945 5954 cpu_manager.go:414] "containerMap: removing container" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910950 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910959 5954 cpu_manager.go:414] "containerMap: removing container" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910972 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910980 5954 cpu_manager.go:414] "containerMap: removing container" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910985 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.910998 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911003 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911008 5954 cpu_manager.go:414] "containerMap: removing container" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911013 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911021 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911026 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911034 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911038 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e8413f-16d8-42b3-a5d1-517199912fc0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911046 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911050 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911056 5954 cpu_manager.go:414] "containerMap: removing container" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911061 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911069 5954 cpu_manager.go:414] "containerMap: removing container" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911073 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911079 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911084 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911089 5954 cpu_manager.go:414] "containerMap: removing container" podUID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911094 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911101 5954 cpu_manager.go:414] "containerMap: removing container" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911121 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911132 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" containerName="payload-pre-processing" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911136 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" containerName="payload-pre-processing" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911149 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911153 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911160 5954 cpu_manager.go:414] "containerMap: removing container" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911164 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911256 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911263 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b7fecc-5c4d-44be-adb0-30654c997e70" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911272 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c47013-4463-4edd-b130-c4b93fea7af9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911279 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ea9a19-fb71-4628-804a-092f6cea7f7e" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911284 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04aa9d5-906e-4480-a073-6c28434e1a5c" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911290 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911296 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed1a6c6-aa68-47c1-a8ae-7d3d1b342269" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911304 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911309 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="31bec522-d1ff-47fc-b8df-b862eb1faae5" containerName="main" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911316 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="06832e97-39ca-478b-91af-70eacec17a09" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911323 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911328 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a64efc-6e67-4742-8542-b6b49ea06f98" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911334 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf6bb35-6c73-4e77-ab15-9eaa15b60ba0" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911339 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f42e1a-5464-4d1f-be9d-6e50c12c9c1e" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911344 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911352 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="840416a8-b1bd-4230-83a4-d6960dbfb0d4" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911357 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="175752d9-a28f-4352-9e81-5dc71de5df45" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911362 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de87552-80de-44bc-9487-058fcd3cbabe" containerName="payload-pre-processing" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911368 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911373 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911380 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911385 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff143bfd-2b06-48c0-8a32-0062ce724b4d" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911391 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd12b01c-dfa7-4a58-8f5d-1a75283dca1f" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911399 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="605626b1-de96-4f7e-83de-02621bca7bbc" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911404 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="78769a77-eb1b-4e93-b3ff-454954858b61" containerName="discovery" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911412 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7188424c-b153-4a5b-86b5-074c5560eba9" containerName="istio-proxy" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911417 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cacbf0a-b64e-4c51-a22b-742e2f2e6924" containerName="maas-api" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911422 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04b177-e9bb-497e-b644-48e0ac2b758b" containerName="cleanup" Nov 03 20:33:40.911708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.911428 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5b16f2-bfeb-4339-9660-32090c2e89ab" containerName="payload-processing" Nov 03 20:33:40.916312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.916287 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:40.922801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.922771 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh"] Nov 03 20:33:40.923273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.923251 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:40.923418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.923397 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:40.977754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.977714 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:40.977930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.977908 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ljt\" (UniqueName: \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:40.978056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:40.978039 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.079587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.079553 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ljt\" (UniqueName: \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.079797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.079696 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.079797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.079772 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.080515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.080478 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.082228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.082206 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.091746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.091722 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ljt\" (UniqueName: \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.240013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.239919 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:41.376392 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.376352 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:41.523610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.523582 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh"] Nov 03 20:33:41.525216 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:33:41.525189 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09563994_8437_460c_a94c_d56b185d341a.slice/crio-59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90 WatchSource:0}: Error finding container 59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90: Status 404 returned error can't find the container with id 59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90 Nov 03 20:33:41.648981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.648945 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" event={"ID":"09563994-8437-460c-a94c-d56b185d341a","Type":"ContainerStarted","Data":"59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90"} Nov 03 20:33:41.969544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:41.969502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.090142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.089999 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.308679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.308641 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:33:42.310090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.310058 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" containerID="cri-o://bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994" gracePeriod=30 Nov 03 20:33:42.312595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.312528 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.428570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.428535 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.615861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.615801 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.683920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.680033 5954 generic.go:356] "Generic (PLEG): container finished" podID="09563994-8437-460c-a94c-d56b185d341a" containerID="55c0d897bbfe08a97009bc52265ee20e2d96584f6d5b60facee2c5db2187fd55" exitCode=0 Nov 03 20:33:42.683920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.680097 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" event={"ID":"09563994-8437-460c-a94c-d56b185d341a","Type":"ContainerDied","Data":"55c0d897bbfe08a97009bc52265ee20e2d96584f6d5b60facee2c5db2187fd55"} Nov 03 20:33:42.890363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.890332 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.985402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:42.985276 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.382696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.382666 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.711958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.711907 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.861764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.861735 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:43.960252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.940774 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\" (UniqueName: \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\") pod \"09563994-8437-460c-a94c-d56b185d341a\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " Nov 03 20:33:43.961066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.960623 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\") pod \"09563994-8437-460c-a94c-d56b185d341a\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " Nov 03 20:33:43.961382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.961039 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca" pod "09563994-8437-460c-a94c-d56b185d341a" (UID: "09563994-8437-460c-a94c-d56b185d341a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:43.987666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.961822 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\") pod \"09563994-8437-460c-a94c-d56b185d341a\" (UID: \"09563994-8437-460c-a94c-d56b185d341a\") " Nov 03 20:33:43.988097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.978075 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.989354 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:43.989292 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09563994-8437-460c-a94c-d56b185d341a-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.053602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.053551 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca" pod "09563994-8437-460c-a94c-d56b185d341a" (UID: "09563994-8437-460c-a94c-d56b185d341a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:44.069399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.068745 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt" pod "09563994-8437-460c-a94c-d56b185d341a" (UID: "09563994-8437-460c-a94c-d56b185d341a"). InnerVolumeSpecName "kube-api-access-z7ljt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:44.096492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.096455 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09563994-8437-460c-a94c-d56b185d341a-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.096871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.096847 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ljt\" (UniqueName: \"kubernetes.io/projected/09563994-8437-460c-a94c-d56b185d341a-kube-api-access-z7ljt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:44.222547 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.222448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.318321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.318282 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.519787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.519713 5954 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:33:44.519787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.519768 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:33:44.705139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.705091 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" Nov 03 20:33:44.705139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.705120 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh" event={"ID":"09563994-8437-460c-a94c-d56b185d341a","Type":"ContainerDied","Data":"59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90"} Nov 03 20:33:44.705340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.705170 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59fa5b3a20cc552de2aef3b8eccc417107a1e93182a35b4ea2b171e575769e90" Nov 03 20:33:44.737756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:44.737718 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.316188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.316144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.715532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.715498 5954 generic.go:356] "Generic (PLEG): container finished" podID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerID="6d20c4f85f833655a5ac2d7ef4a034b0ec3eb9c8270b22e146fd52d4fdcd5e7b" exitCode=0 Nov 03 20:33:45.715745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.715603 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" event={"ID":"b0738e36-983b-47ec-8835-e53d1358dbb0","Type":"ContainerDied","Data":"6d20c4f85f833655a5ac2d7ef4a034b0ec3eb9c8270b22e146fd52d4fdcd5e7b"} Nov 03 20:33:45.717513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.717486 5954 generic.go:356] "Generic (PLEG): container finished" podID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerID="31b340494a692fca9dbceed420d9a11d79484aa1deb41e46ddb9569ef5c75caa" exitCode=0 Nov 03 20:33:45.717722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.717705 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" event={"ID":"9bba8f2d-957e-4988-8483-1a3e579285c2","Type":"ContainerDied","Data":"31b340494a692fca9dbceed420d9a11d79484aa1deb41e46ddb9569ef5c75caa"} Nov 03 20:33:45.795349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.795315 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:33:45.832388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.832359 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:33:45.925046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.924996 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925073 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925150 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925176 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925231 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925250 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925344 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925373 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925405 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925500 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925589 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925636 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925673 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.925882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925719 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.925882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925836 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.926119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925920 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\") pod \"b0738e36-983b-47ec-8835-e53d1358dbb0\" (UID: \"b0738e36-983b-47ec-8835-e53d1358dbb0\") " Nov 03 20:33:45.926119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925973 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.926119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926000 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\") pod \"9bba8f2d-957e-4988-8483-1a3e579285c2\" (UID: \"9bba8f2d-957e-4988-8483-1a3e579285c2\") " Nov 03 20:33:45.926300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925491 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925522 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.925868 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926087 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926265 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926530 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926757 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926782 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926799 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926813 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926826 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8f2d-957e-4988-8483-1a3e579285c2-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926840 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926862 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926880 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.926962 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.927215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.927001 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.928088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.927954 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.928088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.927991 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.928088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.928057 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.928281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.928140 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "kube-api-access-ktmwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.928876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.928850 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "kube-api-access-mslw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.929020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.929000 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:45.929093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.929077 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy" pod "b0738e36-983b-47ec-8835-e53d1358dbb0" (UID: "b0738e36-983b-47ec-8835-e53d1358dbb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.929701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:45.929680 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo" pod "9bba8f2d-957e-4988-8483-1a3e579285c2" (UID: "9bba8f2d-957e-4988-8483-1a3e579285c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:46.028093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028047 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028087 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028096 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028150 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028167 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028176 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslw5\" (UniqueName: \"kubernetes.io/projected/b0738e36-983b-47ec-8835-e53d1358dbb0-kube-api-access-mslw5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028187 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmwd\" (UniqueName: \"kubernetes.io/projected/9bba8f2d-957e-4988-8483-1a3e579285c2-kube-api-access-ktmwd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028196 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028206 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8f2d-957e-4988-8483-1a3e579285c2-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028214 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8f2d-957e-4988-8483-1a3e579285c2-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028222 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0738e36-983b-47ec-8835-e53d1358dbb0-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.028384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.028233 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0738e36-983b-47ec-8835-e53d1358dbb0-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.212040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.211931 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:46.725903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.725866 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" Nov 03 20:33:46.726131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.725870 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb" event={"ID":"b0738e36-983b-47ec-8835-e53d1358dbb0","Type":"ContainerDied","Data":"e2469a5a5d80e3c712e877ec9021535d63815992b64be019bcf06e637f4bd556"} Nov 03 20:33:46.726131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.726002 5954 scope.go:122] "RemoveContainer" containerID="6d20c4f85f833655a5ac2d7ef4a034b0ec3eb9c8270b22e146fd52d4fdcd5e7b" Nov 03 20:33:46.727628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.727594 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" Nov 03 20:33:46.727735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.727603 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v" event={"ID":"9bba8f2d-957e-4988-8483-1a3e579285c2","Type":"ContainerDied","Data":"5ad8772dfd22445809763065bce383edd77fc7a34c992fbc63176609ff541d97"} Nov 03 20:33:46.740936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.740911 5954 scope.go:122] "RemoveContainer" containerID="31b340494a692fca9dbceed420d9a11d79484aa1deb41e46ddb9569ef5c75caa" Nov 03 20:33:46.761143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.760269 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:33:46.764619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.764588 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-sfgfb"] Nov 03 20:33:46.781235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.781192 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:33:46.783790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:46.783763 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-szp8v"] Nov 03 20:33:47.007566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.007474 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:47.188590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.188561 5954 scope.go:122] "RemoveContainer" containerID="e0f289509ba14dac3dbe99ab2c6101345fd7376e01d619ae33a88b2e58a92197" Nov 03 20:33:47.199439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.199417 5954 scope.go:122] "RemoveContainer" containerID="301e46984ef0c7e56ec73043ba458c9f34d8f6367b807d426f1c8ae6d7a2800b" Nov 03 20:33:47.405481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.405452 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" path="/var/lib/kubelet/pods/9bba8f2d-957e-4988-8483-1a3e579285c2/volumes" Nov 03 20:33:47.405995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.405980 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" path="/var/lib/kubelet/pods/b0738e36-983b-47ec-8835-e53d1358dbb0/volumes" Nov 03 20:33:47.624569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.624476 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:33:47.737328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.737295 5954 generic.go:356] "Generic (PLEG): container finished" podID="e2fb5d90-f035-41ea-b267-3434641359d4" containerID="bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994" exitCode=0 Nov 03 20:33:47.737541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.737365 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" Nov 03 20:33:47.737541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.737380 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" event={"ID":"e2fb5d90-f035-41ea-b267-3434641359d4","Type":"ContainerDied","Data":"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994"} Nov 03 20:33:47.737541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.737459 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z" event={"ID":"e2fb5d90-f035-41ea-b267-3434641359d4","Type":"ContainerDied","Data":"53e74269a694993e1399b4dca88bbc63d12dbe6dd9ea7f87034321c1e854dc8d"} Nov 03 20:33:47.737541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.737488 5954 scope.go:122] "RemoveContainer" containerID="bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994" Nov 03 20:33:47.747729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.747707 5954 scope.go:122] "RemoveContainer" containerID="bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994" Nov 03 20:33:47.748036 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:33:47.748017 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994\": container with ID starting with bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994 not found: ID does not exist" containerID="bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994" Nov 03 20:33:47.748123 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.748051 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994"} err="failed to get container status \"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994\": rpc error: code = NotFound desc = could not find container \"bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994\": container with ID starting with bf8178ddd87ff200eef3b7a2bf4d2d875d128eb8c17fc222691c9b8941d53994 not found: ID does not exist" Nov 03 20:33:47.771348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771314 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771414 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771471 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771521 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771581 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771596 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.771822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771724 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.771822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771742 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.771822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771734 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771825 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771869 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.771998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.771897 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\") pod \"e2fb5d90-f035-41ea-b267-3434641359d4\" (UID: \"e2fb5d90-f035-41ea-b267-3434641359d4\") " Nov 03 20:33:47.772165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772089 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.772576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772550 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.772686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772581 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.772686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772585 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:47.772686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772596 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.772686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.772630 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.773855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.773825 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.773959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.773884 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.773959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.773932 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:47.774251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.774230 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s" pod "e2fb5d90-f035-41ea-b267-3434641359d4" (UID: "e2fb5d90-f035-41ea-b267-3434641359d4"). InnerVolumeSpecName "kube-api-access-7729s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.874078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.874043 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fb5d90-f035-41ea-b267-3434641359d4-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.874078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.874077 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.874078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.874086 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fb5d90-f035-41ea-b267-3434641359d4-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.874351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.874095 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fb5d90-f035-41ea-b267-3434641359d4-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.874351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.874118 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7729s\" (UniqueName: \"kubernetes.io/projected/e2fb5d90-f035-41ea-b267-3434641359d4-kube-api-access-7729s\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.975594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:47.975559 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:48.071923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:48.071890 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:33:48.078457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:48.078426 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-b6d6z"] Nov 03 20:33:48.693640 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:48.693606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:48.828818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:48.828782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.381291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:49.381261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.415928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:49.415896 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" path="/var/lib/kubelet/pods/e2fb5d90-f035-41ea-b267-3434641359d4/volumes" Nov 03 20:33:49.897658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:49.897611 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.997610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:49.997518 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:50.185992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:50.185898 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:50.284991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:50.284959 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.148787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:51.148738 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.246943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:51.246912 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.591007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:51.590920 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.696009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:51.695978 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.912198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:51.912168 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.195472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.195391 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt"] Nov 03 20:33:52.195984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.195967 5954 cpu_manager.go:414] "containerMap: removing container" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.195984 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.195997 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196003 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196019 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196026 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196038 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" Nov 03 20:33:52.196050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196045 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196060 5954 cpu_manager.go:414] "containerMap: removing container" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196067 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33533d2a-ed7c-48a1-9fbb-aaaf2264b3ee" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196078 5954 cpu_manager.go:414] "containerMap: removing container" podUID="09563994-8437-460c-a94c-d56b185d341a" containerName="revoke-keys" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196084 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09563994-8437-460c-a94c-d56b185d341a" containerName="revoke-keys" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196209 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="16787376-4114-4876-8e0a-cd916a455f71" containerName="maas-api" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196217 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0738e36-983b-47ec-8835-e53d1358dbb0" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196224 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="610daeb7-56b0-495e-945c-92b44d93709d" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196233 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bba8f2d-957e-4988-8483-1a3e579285c2" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196240 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6761899d-1c05-4ecb-8373-27c9c5813ac0" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196245 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2fb5d90-f035-41ea-b267-3434641359d4" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196252 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="09563994-8437-460c-a94c-d56b185d341a" containerName="revoke-keys" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196258 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d7cb77-77fe-4bf2-bef6-b0374b2f17cd" containerName="istio-proxy" Nov 03 20:33:52.196319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.196265 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f6a4cc-ae68-474e-81ba-7e1675060159" containerName="istio-proxy" Nov 03 20:33:52.201435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.201405 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.205786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.205761 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:52.206077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.206046 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:52.208172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.208079 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt"] Nov 03 20:33:52.239190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.239145 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.239405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.239205 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cv9\" (UniqueName: \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.239484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.239404 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.340937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.340896 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.340937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.340944 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cv9\" (UniqueName: \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.341277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.341055 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.341670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.341648 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.343371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.343350 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.353301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.353273 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cv9\" (UniqueName: \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\") pod \"maas-api-revoke-sub-c04d0a2f-phcwt\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.519286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.519197 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:52.592991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.592954 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.803274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.803248 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt"] Nov 03 20:33:52.805501 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:33:52.805459 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6759f67_5884_4bf9_a11c_d7abbf7aea70.slice/crio-f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c WatchSource:0}: Error finding container f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c: Status 404 returned error can't find the container with id f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c Nov 03 20:33:52.816248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:52.816214 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" event={"ID":"f6759f67-5884-4bf9-a11c-d7abbf7aea70","Type":"ContainerStarted","Data":"f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c"} Nov 03 20:33:53.275362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:53.275321 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:53.824043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:53.824009 5954 generic.go:356] "Generic (PLEG): container finished" podID="f6759f67-5884-4bf9-a11c-d7abbf7aea70" containerID="4b60f35ff90031fb3695aeaf42522e5d5f7b31ea51763f542fe59f7b85fdb574" exitCode=0 Nov 03 20:33:53.824257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:53.824095 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" event={"ID":"f6759f67-5884-4bf9-a11c-d7abbf7aea70","Type":"ContainerDied","Data":"4b60f35ff90031fb3695aeaf42522e5d5f7b31ea51763f542fe59f7b85fdb574"} Nov 03 20:33:53.996748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:53.996655 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.695027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:54.694992 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.983735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:54.983709 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:55.081923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.081890 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\") pod \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " Nov 03 20:33:55.082161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.081985 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\") pod \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " Nov 03 20:33:55.082248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.082207 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\" (UniqueName: \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\") pod \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\" (UID: \"f6759f67-5884-4bf9-a11c-d7abbf7aea70\") " Nov 03 20:33:55.082326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.082302 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca" pod "f6759f67-5884-4bf9-a11c-d7abbf7aea70" (UID: "f6759f67-5884-4bf9-a11c-d7abbf7aea70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:55.082727 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.082699 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.084239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.084205 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca" pod "f6759f67-5884-4bf9-a11c-d7abbf7aea70" (UID: "f6759f67-5884-4bf9-a11c-d7abbf7aea70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:55.084356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.084323 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9" pod "f6759f67-5884-4bf9-a11c-d7abbf7aea70" (UID: "f6759f67-5884-4bf9-a11c-d7abbf7aea70"). InnerVolumeSpecName "kube-api-access-n9cv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:55.183904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.183852 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9cv9\" (UniqueName: \"kubernetes.io/projected/f6759f67-5884-4bf9-a11c-d7abbf7aea70-kube-api-access-n9cv9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.183904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.183902 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6759f67-5884-4bf9-a11c-d7abbf7aea70-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.621145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.610777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:55.838199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.838085 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" Nov 03 20:33:55.838199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.838127 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt" event={"ID":"f6759f67-5884-4bf9-a11c-d7abbf7aea70","Type":"ContainerDied","Data":"f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c"} Nov 03 20:33:55.838639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:55.838204 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7f9e55ba309f7a56e7e806f1abfda936972621723f0b44d4ea144984628d13c" Nov 03 20:33:56.399505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:56.399466 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.169443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:57.169409 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.843702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:57.843669 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.589502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:58.589467 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:59.396504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:33:59.396470 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.023617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.023580 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.206677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.206639 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.308421 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:00.308317 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:34:00.465230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.465190 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.487468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.487423 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh"] Nov 03 20:34:00.490500 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.490468 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b35c30-8sxsh"] Nov 03 20:34:00.578825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.578726 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.999719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:00.999682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.100829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.100787 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.405908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.405873 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09563994-8437-460c-a94c-d56b185d341a" path="/var/lib/kubelet/pods/09563994-8437-460c-a94c-d56b185d341a/volumes" Nov 03 20:34:01.476908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.476881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.914411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.914376 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:34:01.914717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.914675 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" containerID="cri-o://2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3" gracePeriod=30 Nov 03 20:34:01.942741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:01.942685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.123371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:02.114676 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.337755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:02.337650 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.444296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:02.444262 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.570651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:02.570610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.990408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:02.990374 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:03.125721 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:03.124956 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:03.593915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:03.593877 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:04.683736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:04.683700 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:05.587028 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:05.586994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.391838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:06.391804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.988316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:06.988285 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:34:07.202790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.202763 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:34:07.278502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278470 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278546 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278648 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278691 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278726 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278753 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278840 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278868 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.278984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.278935 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:07.279195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279058 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\") pod \"fe622415-b8a8-4fe5-a9a6-d796454de989\" (UID: \"fe622415-b8a8-4fe5-a9a6-d796454de989\") " Nov 03 20:34:07.279195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279068 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:07.279318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279222 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:07.279318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279239 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:07.279501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279329 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:07.279857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279759 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe622415-b8a8-4fe5-a9a6-d796454de989-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.279857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279789 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.279857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279804 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.279857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279818 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.279857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.279832 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.281069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.281036 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:07.281191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.281131 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:07.281263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.281239 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "kube-api-access-qztj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:07.281818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.281799 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo" pod "fe622415-b8a8-4fe5-a9a6-d796454de989" (UID: "fe622415-b8a8-4fe5-a9a6-d796454de989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:07.285940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.285917 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:07.381465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.381430 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.381465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.381460 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztj9\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-kube-api-access-qztj9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.381465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.381472 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.381704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.381481 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe622415-b8a8-4fe5-a9a6-d796454de989-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.915654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.915612 5954 generic.go:356] "Generic (PLEG): container finished" podID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerID="2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3" exitCode=0 Nov 03 20:34:07.916193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.915669 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" Nov 03 20:34:07.916193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.915700 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerDied","Data":"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3"} Nov 03 20:34:07.916193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.915762 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d" event={"ID":"fe622415-b8a8-4fe5-a9a6-d796454de989","Type":"ContainerDied","Data":"e1141d75ccae6b06721ffa04285063d1ef75a8dd149c89912b0b81f87a318083"} Nov 03 20:34:07.916193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.915788 5954 scope.go:122] "RemoveContainer" containerID="2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3" Nov 03 20:34:07.926369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.926332 5954 scope.go:122] "RemoveContainer" containerID="c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11" Nov 03 20:34:07.936529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.936504 5954 scope.go:122] "RemoveContainer" containerID="2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3" Nov 03 20:34:07.936857 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:07.936835 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3\": container with ID starting with 2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3 not found: ID does not exist" containerID="2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3" Nov 03 20:34:07.936914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.936866 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3"} err="failed to get container status \"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3\": rpc error: code = NotFound desc = could not find container \"2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3\": container with ID starting with 2d08865a54d2289d6f7a7e9b1c0f8a9540fa44dac3fee8a273b41f87d8ad66e3 not found: ID does not exist" Nov 03 20:34:07.936914 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.936885 5954 scope.go:122] "RemoveContainer" containerID="c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11" Nov 03 20:34:07.937213 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:07.937192 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11\": container with ID starting with c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11 not found: ID does not exist" containerID="c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11" Nov 03 20:34:07.937292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.937227 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11"} err="failed to get container status \"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11\": rpc error: code = NotFound desc = could not find container \"c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11\": container with ID starting with c7131c88f0158fd4d9f76ac9fd053d411b9cbce4153daa3a4d7dc4d034e59c11 not found: ID does not exist" Nov 03 20:34:07.937381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.937341 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:34:07.943102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:07.943080 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b0adda-openshift-default-75774c8877-lc65d"] Nov 03 20:34:08.049192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:08.049153 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.160754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:08.160719 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.275408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:08.275318 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.466471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:08.445981 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.405559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:09.405527 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" path="/var/lib/kubelet/pods/fe622415-b8a8-4fe5-a9a6-d796454de989/volumes" Nov 03 20:34:09.466411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:09.466378 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.091321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:10.091281 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.186825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:10.186777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.397423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:10.397387 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.519669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:10.519633 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.892162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:10.892089 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:11.444947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:11.444919 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.325400 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:12.325369 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.743689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:12.743652 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.854239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:12.854204 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:13.217185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:13.217145 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.043386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:14.043352 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.187137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:14.180211 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:15.180884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:15.180850 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:15.853808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:15.853770 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:34:15.867434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:15.867404 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:34:15.966726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:15.966691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:16.190746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:16.190649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.188415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:17.188367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:18.283131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:18.279937 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.075410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:19.075354 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.187887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:19.187847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.289621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:19.289587 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.974101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:19.974059 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:20.886643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:20.886599 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:21.348428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:21.348343 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:34:21.412216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:21.412183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:21.511985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:21.511921 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:21.812798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:21.812733 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.404542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:22.404500 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.534588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:22.534558 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.630781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:22.630747 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.005982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:23.005949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.106285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:23.106250 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.206812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:23.206776 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.567699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:23.567664 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.786184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:23.786148 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.483670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:24.483634 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.599630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:24.599594 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.694449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:24.694414 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:25.271973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:25.271940 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:25.935830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:25.935784 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.349312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:26.349210 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" containerID="cri-o://1658dfc4fa260bf3c7db7357279800e6afb6b706ea5e2f738c9afd88dc09b86b" gracePeriod=25 Nov 03 20:34:26.677207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:26.677172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:27.474716 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:27.474685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:27.979652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:27.979613 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:28.310322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:28.310238 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:29.272608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:29.272564 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:29.832822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:29.832783 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:34:29.833151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:29.833038 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" podUID="49fc622d-4e5d-4d9e-a2db-625b225afeca" containerName="maas-api" containerID="cri-o://e8515ad42c99bb940f620e9dfb7079eca4144f7d3a53c677447c9acd8b659e2b" gracePeriod=30 Nov 03 20:34:29.879294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:29.879261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.039136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.034008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.180465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.180425 5954 generic.go:356] "Generic (PLEG): container finished" podID="49fc622d-4e5d-4d9e-a2db-625b225afeca" containerID="e8515ad42c99bb940f620e9dfb7079eca4144f7d3a53c677447c9acd8b659e2b" exitCode=0 Nov 03 20:34:30.180974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.180736 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" event={"ID":"49fc622d-4e5d-4d9e-a2db-625b225afeca","Type":"ContainerDied","Data":"e8515ad42c99bb940f620e9dfb7079eca4144f7d3a53c677447c9acd8b659e2b"} Nov 03 20:34:30.223247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.223220 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:34:30.264069 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:30.264033 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:34:30.293873 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.293839 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.331919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.331881 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\") pod \"49fc622d-4e5d-4d9e-a2db-625b225afeca\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " Nov 03 20:34:30.332148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.331982 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\") pod \"49fc622d-4e5d-4d9e-a2db-625b225afeca\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " Nov 03 20:34:30.332239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.332168 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\" (UniqueName: \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\") pod \"49fc622d-4e5d-4d9e-a2db-625b225afeca\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " Nov 03 20:34:30.332239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.332211 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\") pod \"49fc622d-4e5d-4d9e-a2db-625b225afeca\" (UID: \"49fc622d-4e5d-4d9e-a2db-625b225afeca\") " Nov 03 20:34:30.332574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.332542 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca" pod "49fc622d-4e5d-4d9e-a2db-625b225afeca" (UID: "49fc622d-4e5d-4d9e-a2db-625b225afeca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:30.333195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.333172 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fc622d-4e5d-4d9e-a2db-625b225afeca-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:30.334410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.334386 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls" pod "49fc622d-4e5d-4d9e-a2db-625b225afeca" (UID: "49fc622d-4e5d-4d9e-a2db-625b225afeca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:30.334698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.334672 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6" pod "49fc622d-4e5d-4d9e-a2db-625b225afeca" (UID: "49fc622d-4e5d-4d9e-a2db-625b225afeca"). InnerVolumeSpecName "kube-api-access-5vmx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:30.334818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.334733 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs" pod "49fc622d-4e5d-4d9e-a2db-625b225afeca" (UID: "49fc622d-4e5d-4d9e-a2db-625b225afeca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:30.395628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.395594 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.434485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.434402 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:30.434485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.434431 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fc622d-4e5d-4d9e-a2db-625b225afeca-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:30.434485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.434444 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmx6\" (UniqueName: \"kubernetes.io/projected/49fc622d-4e5d-4d9e-a2db-625b225afeca-kube-api-access-5vmx6\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:30.502524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.502474 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.755092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.755006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.891132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.875287 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.898129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.893339 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" containerID="cri-o://73c946123f596bc156e83b4c84d7c4434c036e75364c19251dadcc22abc1d6ca" gracePeriod=45 Nov 03 20:34:30.914233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:30.914177 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" containerID="cri-o://45fff91e862076de240ddb5f5256b7c9edd2821cac7c06ce6e5d2809fa0ab833" gracePeriod=45 Nov 03 20:34:31.012149 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.012062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.127096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.123233 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.191523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.191482 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" event={"ID":"49fc622d-4e5d-4d9e-a2db-625b225afeca","Type":"ContainerDied","Data":"1f4aba128e8774bfa1a4af9d8b1a4bdb624f4447ede55d6b7616997320ec287e"} Nov 03 20:34:31.191689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.191496 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb" Nov 03 20:34:31.191781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.191616 5954 scope.go:122] "RemoveContainer" containerID="e8515ad42c99bb940f620e9dfb7079eca4144f7d3a53c677447c9acd8b659e2b" Nov 03 20:34:31.195321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.195289 5954 generic.go:356] "Generic (PLEG): container finished" podID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerID="73c946123f596bc156e83b4c84d7c4434c036e75364c19251dadcc22abc1d6ca" exitCode=0 Nov 03 20:34:31.195571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.195510 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" event={"ID":"cb619c24-a539-4182-8f1c-2c72af9f106e","Type":"ContainerDied","Data":"73c946123f596bc156e83b4c84d7c4434c036e75364c19251dadcc22abc1d6ca"} Nov 03 20:34:31.198046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.198019 5954 generic.go:356] "Generic (PLEG): container finished" podID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerID="45fff91e862076de240ddb5f5256b7c9edd2821cac7c06ce6e5d2809fa0ab833" exitCode=0 Nov 03 20:34:31.198199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.198046 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" event={"ID":"7b0eb303-045b-47a0-a42a-4fd19eecc3ed","Type":"ContainerDied","Data":"45fff91e862076de240ddb5f5256b7c9edd2821cac7c06ce6e5d2809fa0ab833"} Nov 03 20:34:31.221984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.221953 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.292946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.292861 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:34:31.311281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.310969 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:34:31.318333 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.318297 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5"] Nov 03 20:34:31.319385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319353 5954 cpu_manager.go:414] "containerMap: removing container" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:31.319385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319382 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319451 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f6759f67-5884-4bf9-a11c-d7abbf7aea70" containerName="revoke-keys" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319462 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6759f67-5884-4bf9-a11c-d7abbf7aea70" containerName="revoke-keys" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319500 5954 cpu_manager.go:414] "containerMap: removing container" podUID="49fc622d-4e5d-4d9e-a2db-625b225afeca" containerName="maas-api" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319510 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fc622d-4e5d-4d9e-a2db-625b225afeca" containerName="maas-api" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319560 5954 cpu_manager.go:414] "containerMap: removing container" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" Nov 03 20:34:31.319601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319570 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" Nov 03 20:34:31.319923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319726 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:31.319923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319739 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6759f67-5884-4bf9-a11c-d7abbf7aea70" containerName="revoke-keys" Nov 03 20:34:31.319923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319754 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fc622d-4e5d-4d9e-a2db-625b225afeca" containerName="maas-api" Nov 03 20:34:31.319923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319771 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:31.319923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.319779 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" containerName="main" Nov 03 20:34:31.325590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.325558 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.329746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.329698 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:34:31.330044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.330019 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:31.330195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.330088 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:31.333206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.333180 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c2c601-5589c47587-sbvhb"] Nov 03 20:34:31.337280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.337260 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5"] Nov 03 20:34:31.342987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.342436 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:34:31.342987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.342644 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" containerID="cri-o://cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807" gracePeriod=30 Nov 03 20:34:31.402927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.402892 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.410177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.408929 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fc622d-4e5d-4d9e-a2db-625b225afeca" path="/var/lib/kubelet/pods/49fc622d-4e5d-4d9e-a2db-625b225afeca/volumes" Nov 03 20:34:31.474881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.474842 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.475098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.474963 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.475098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475011 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\" (UniqueName: \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.475098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475064 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.475327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475296 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.475379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475141 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.475429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475413 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.475496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475459 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.475549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475519 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.475602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475582 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\" (UniqueName: \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475665 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475740 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\") pod \"cb619c24-a539-4182-8f1c-2c72af9f106e\" (UID: \"cb619c24-a539-4182-8f1c-2c72af9f106e\") " Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.475777 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\") pod \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\" (UID: \"7b0eb303-045b-47a0-a42a-4fd19eecc3ed\") " Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476130 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476247 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476045 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476332 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476346 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476506 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzwz\" (UniqueName: \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476632 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476651 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476666 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.476718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476134 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.477565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476763 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.477565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.476779 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.478095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.478001 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:31.478095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.478060 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "kube-api-access-xfmg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:31.478911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.478884 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.479405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.479380 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs" pod "cb619c24-a539-4182-8f1c-2c72af9f106e" (UID: "cb619c24-a539-4182-8f1c-2c72af9f106e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:31.479734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.479709 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "kube-api-access-dptfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:31.479853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.479825 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm" pod "7b0eb303-045b-47a0-a42a-4fd19eecc3ed" (UID: "7b0eb303-045b-47a0-a42a-4fd19eecc3ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.578079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.577971 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.578327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578121 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzwz\" (UniqueName: \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.578447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578364 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578460 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578477 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb619c24-a539-4182-8f1c-2c72af9f106e-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578490 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578504 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptfl\" (UniqueName: \"kubernetes.io/projected/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-kube-api-access-dptfl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578519 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb619c24-a539-4182-8f1c-2c72af9f106e-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578532 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmg8\" (UniqueName: \"kubernetes.io/projected/cb619c24-a539-4182-8f1c-2c72af9f106e-kube-api-access-xfmg8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.578543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578545 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.579004 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.578558 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0eb303-045b-47a0-a42a-4fd19eecc3ed-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.579312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.579287 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.582459 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.582410 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.643169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.618762 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzwz\" (UniqueName: \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.651142 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.649384 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:31.871275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.871239 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.947130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:31.947092 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5"] Nov 03 20:34:31.949125 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:34:31.949082 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa11a05_5aaf_4374_b4f0_6ea56151c1ac.slice/crio-6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40 WatchSource:0}: Error finding container 6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40: Status 404 returned error can't find the container with id 6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40 Nov 03 20:34:32.028137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028087 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb"] Nov 03 20:34:32.028732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028715 5954 cpu_manager.go:414] "containerMap: removing container" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:32.028789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028734 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe622415-b8a8-4fe5-a9a6-d796454de989" containerName="istio-proxy" Nov 03 20:34:32.028789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028781 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" Nov 03 20:34:32.028789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028787 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" Nov 03 20:34:32.028916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.028896 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" containerName="main" Nov 03 20:34:32.033489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.033468 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.041427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.041401 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb"] Nov 03 20:34:32.188144 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.188086 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzckx\" (UniqueName: \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.188297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.188224 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.188297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.188263 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.205286 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.205252 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" event={"ID":"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac","Type":"ContainerStarted","Data":"6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40"} Nov 03 20:34:32.207581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.207556 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" Nov 03 20:34:32.207581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.207570 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5" event={"ID":"cb619c24-a539-4182-8f1c-2c72af9f106e","Type":"ContainerDied","Data":"15def84ad78a1c3ae7802529012f4244182ef7e8bf8fdd8d65c700c377437c7f"} Nov 03 20:34:32.207800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.207621 5954 scope.go:122] "RemoveContainer" containerID="73c946123f596bc156e83b4c84d7c4434c036e75364c19251dadcc22abc1d6ca" Nov 03 20:34:32.209031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.209011 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" event={"ID":"7b0eb303-045b-47a0-a42a-4fd19eecc3ed","Type":"ContainerDied","Data":"ec67e0add7fa62c8670471e18485adde54096200a54196b3a94be4e7c44b324b"} Nov 03 20:34:32.209157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.209056 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns" Nov 03 20:34:32.230622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.230599 5954 scope.go:122] "RemoveContainer" containerID="45fff91e862076de240ddb5f5256b7c9edd2821cac7c06ce6e5d2809fa0ab833" Nov 03 20:34:32.241397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.241374 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:34:32.244796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.244771 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-2-w3-62221d-kserve-6dc8fb5d8-grfh5"] Nov 03 20:34:32.257298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.257267 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:34:32.263085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.263059 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/premium-w3-62221d-kserve-5dd9745dd4-kk5ns"] Nov 03 20:34:32.289293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.289263 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzckx\" (UniqueName: \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.289432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.289343 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.289432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.289386 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.290170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.290142 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.291758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.291739 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.301530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.301505 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzckx\" (UniqueName: \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\") pod \"maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.357020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.356993 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:32.548951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.548915 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:32.652984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:32.652951 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb"] Nov 03 20:34:32.655048 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:34:32.655015 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd356c513_0be7_42ff_88f3_7a3d0a32b581.slice/crio-d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782 WatchSource:0}: Error finding container d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782: Status 404 returned error can't find the container with id d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782 Nov 03 20:34:33.219598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.219491 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" event={"ID":"d356c513-0be7-42ff-88f3-7a3d0a32b581","Type":"ContainerStarted","Data":"672ef62045193370d3a341e037cc79f907aca3b8752a219b644f74b7ca597d8b"} Nov 03 20:34:33.219598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.219557 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" event={"ID":"d356c513-0be7-42ff-88f3-7a3d0a32b581","Type":"ContainerStarted","Data":"d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782"} Nov 03 20:34:33.221979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.221953 5954 generic.go:356] "Generic (PLEG): container finished" podID="7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" containerID="d044a8639a6475613847ee60345ad78e32b43b217047edbb02923c97dd526291" exitCode=0 Nov 03 20:34:33.222086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.222006 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" event={"ID":"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac","Type":"ContainerDied","Data":"d044a8639a6475613847ee60345ad78e32b43b217047edbb02923c97dd526291"} Nov 03 20:34:33.240685 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.240638 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" podStartSLOduration=1.2406220239999999 podStartE2EDuration="1.240622024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:33.238165743 +0000 UTC m=+2213.963173339" watchObservedRunningTime="2026-11-03 20:34:33.240622024 +0000 UTC m=+2213.965629619" Nov 03 20:34:33.382867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.382822 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:33.406445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.406416 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0eb303-045b-47a0-a42a-4fd19eecc3ed" path="/var/lib/kubelet/pods/7b0eb303-045b-47a0-a42a-4fd19eecc3ed/volumes" Nov 03 20:34:33.406899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:33.406884 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb619c24-a539-4182-8f1c-2c72af9f106e" path="/var/lib/kubelet/pods/cb619c24-a539-4182-8f1c-2c72af9f106e/volumes" Nov 03 20:34:34.166532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.166491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.248450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.248414 5954 generic.go:356] "Generic (PLEG): container finished" podID="d356c513-0be7-42ff-88f3-7a3d0a32b581" containerID="672ef62045193370d3a341e037cc79f907aca3b8752a219b644f74b7ca597d8b" exitCode=0 Nov 03 20:34:34.248668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.248589 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" event={"ID":"d356c513-0be7-42ff-88f3-7a3d0a32b581","Type":"ContainerDied","Data":"672ef62045193370d3a341e037cc79f907aca3b8752a219b644f74b7ca597d8b"} Nov 03 20:34:34.431981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.431918 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:34.457093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.457055 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\") pod \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " Nov 03 20:34:34.457371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.457217 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\" (UniqueName: \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\") pod \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " Nov 03 20:34:34.457455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.457396 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\") pod \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\" (UID: \"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac\") " Nov 03 20:34:34.457986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.457957 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca" pod "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" (UID: "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:34.460461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.460435 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz" pod "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" (UID: "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac"). InnerVolumeSpecName "kube-api-access-ltzwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:34.461035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.461008 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca" pod "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" (UID: "7aa11a05-5aaf-4374-b4f0-6ea56151c1ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:34.559055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.559019 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzwz\" (UniqueName: \"kubernetes.io/projected/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-kube-api-access-ltzwz\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:34.559055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.559054 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:34.559308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:34.559068 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:35.141167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.141078 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:35.256366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.256338 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" Nov 03 20:34:35.256554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.256373 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5" event={"ID":"7aa11a05-5aaf-4374-b4f0-6ea56151c1ac","Type":"ContainerDied","Data":"6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40"} Nov 03 20:34:35.256554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.256433 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cfaf1467585e776706a7ed695415acc0fab0d08d9f547ba6d7ad582b2d82f40" Nov 03 20:34:35.535782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.535749 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:35.622925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.622891 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\") pod \"d356c513-0be7-42ff-88f3-7a3d0a32b581\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " Nov 03 20:34:35.623134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.623016 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\" (UniqueName: \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\") pod \"d356c513-0be7-42ff-88f3-7a3d0a32b581\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " Nov 03 20:34:35.623243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.623227 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\") pod \"d356c513-0be7-42ff-88f3-7a3d0a32b581\" (UID: \"d356c513-0be7-42ff-88f3-7a3d0a32b581\") " Nov 03 20:34:35.623603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.623569 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca" pod "d356c513-0be7-42ff-88f3-7a3d0a32b581" (UID: "d356c513-0be7-42ff-88f3-7a3d0a32b581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:35.623733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.623721 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:35.625118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.625080 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca" pod "d356c513-0be7-42ff-88f3-7a3d0a32b581" (UID: "d356c513-0be7-42ff-88f3-7a3d0a32b581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:35.625224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.625120 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx" pod "d356c513-0be7-42ff-88f3-7a3d0a32b581" (UID: "d356c513-0be7-42ff-88f3-7a3d0a32b581"). InnerVolumeSpecName "kube-api-access-kzckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:35.724684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.724647 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d356c513-0be7-42ff-88f3-7a3d0a32b581-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:35.724684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.724680 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzckx\" (UniqueName: \"kubernetes.io/projected/d356c513-0be7-42ff-88f3-7a3d0a32b581-kube-api-access-kzckx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:35.978357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:35.978325 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:36.264994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.264904 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" Nov 03 20:34:36.264994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.264935 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb" event={"ID":"d356c513-0be7-42ff-88f3-7a3d0a32b581","Type":"ContainerDied","Data":"d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782"} Nov 03 20:34:36.264994 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.264993 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52dac16d7fbbd3a16637e46d0e91f0e7c2b8088fb7c3de828a762f0396fc782" Nov 03 20:34:36.612035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.612005 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:34:36.737689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737656 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.737916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737751 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.737916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737795 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.737916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737854 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.737916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737884 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.738153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.737977 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.738153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738141 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.738262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738225 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.738320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738281 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\") pod \"ff04bd71-316c-473d-a766-220368ec68f4\" (UID: \"ff04bd71-316c-473d-a766-220368ec68f4\") " Nov 03 20:34:36.738320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738288 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.738320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738304 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.738508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738488 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.738730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738708 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.738789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738730 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:36.738905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738886 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.738969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738913 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff04bd71-316c-473d-a766-220368ec68f4-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.738969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738930 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.738969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738945 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.738969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.738963 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.740275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.740250 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "kube-api-access-cr548". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:36.740488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.740458 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:36.741093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.741070 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:36.741236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.741188 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy" pod "ff04bd71-316c-473d-a766-220368ec68f4" (UID: "ff04bd71-316c-473d-a766-220368ec68f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:36.779989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.779904 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:36.841273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.841226 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.841465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.841280 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd71-316c-473d-a766-220368ec68f4-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.841465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.841300 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr548\" (UniqueName: \"kubernetes.io/projected/ff04bd71-316c-473d-a766-220368ec68f4-kube-api-access-cr548\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.841465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.841316 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff04bd71-316c-473d-a766-220368ec68f4-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:36.885098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.885059 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:34:36.986698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:36.986662 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.145365 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.145317 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.250838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.250497 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.285268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.285223 5954 generic.go:356] "Generic (PLEG): container finished" podID="ff04bd71-316c-473d-a766-220368ec68f4" containerID="cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807" exitCode=0 Nov 03 20:34:37.285482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.285426 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerDied","Data":"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807"} Nov 03 20:34:37.285555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.285507 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" event={"ID":"ff04bd71-316c-473d-a766-220368ec68f4","Type":"ContainerDied","Data":"680c0a89ac39c26d84d0d360dcef87eb3032c493f3f54f468a39b1b2f8903ce2"} Nov 03 20:34:37.285555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.285535 5954 scope.go:122] "RemoveContainer" containerID="cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807" Nov 03 20:34:37.286773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.285677 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt" Nov 03 20:34:37.304423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.304399 5954 scope.go:122] "RemoveContainer" containerID="ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35" Nov 03 20:34:37.319665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.319642 5954 scope.go:122] "RemoveContainer" containerID="cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807" Nov 03 20:34:37.320137 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:37.320091 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807\": container with ID starting with cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807 not found: ID does not exist" containerID="cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807" Nov 03 20:34:37.321019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.320978 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807"} err="failed to get container status \"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807\": rpc error: code = NotFound desc = could not find container \"cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807\": container with ID starting with cebcf7fcadd3862a92330def2ffd727322b059338ac82a09d2f21b87c3b6c807 not found: ID does not exist" Nov 03 20:34:37.321166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.321022 5954 scope.go:122] "RemoveContainer" containerID="ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35" Nov 03 20:34:37.322305 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:37.321759 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35\": container with ID starting with ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35 not found: ID does not exist" containerID="ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35" Nov 03 20:34:37.322305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.321800 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35"} err="failed to get container status \"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35\": rpc error: code = NotFound desc = could not find container \"ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35\": container with ID starting with ca0263ea4835bf6a7845fa479834c672007896c617e302e9f685606101641b35 not found: ID does not exist" Nov 03 20:34:37.325589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.325564 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:34:37.329891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.329863 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ee4c78-openshift-default-5b6f9dfccc-7dzbt"] Nov 03 20:34:37.404869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.404792 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff04bd71-316c-473d-a766-220368ec68f4" path="/var/lib/kubelet/pods/ff04bd71-316c-473d-a766-220368ec68f4/volumes" Nov 03 20:34:37.460655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.460622 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:34:37.460843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.460820 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" containerID="cri-o://c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9" gracePeriod=30 Nov 03 20:34:37.500437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.500404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.800589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.800499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.979549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:37.979507 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:38.789189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:38.789154 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:39.771856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:39.771820 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:40.774443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:40.774411 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.486406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:41.486372 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.586402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:41.586367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.965606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:41.965576 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:42.166213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:42.166182 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:34:42.179606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:42.179572 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:34:42.240068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:42.239987 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:42.340774 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:42.340742 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:43.016376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.016348 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:43.218930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.218896 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:34:43.327181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.327074 5954 generic.go:356] "Generic (PLEG): container finished" podID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerID="c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9" exitCode=0 Nov 03 20:34:43.327181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.327162 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" Nov 03 20:34:43.327373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.327164 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" event={"ID":"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c","Type":"ContainerDied","Data":"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9"} Nov 03 20:34:43.327373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.327280 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t" event={"ID":"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c","Type":"ContainerDied","Data":"54210336b0da5b5579abe13b9b62026f80e5d0272e4fe14023b59473be7baf3f"} Nov 03 20:34:43.327373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.327304 5954 scope.go:122] "RemoveContainer" containerID="c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9" Nov 03 20:34:43.337800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.337776 5954 scope.go:122] "RemoveContainer" containerID="c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9" Nov 03 20:34:43.338059 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:43.338043 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9\": container with ID starting with c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9 not found: ID does not exist" containerID="c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9" Nov 03 20:34:43.338132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.338067 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9"} err="failed to get container status \"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9\": rpc error: code = NotFound desc = could not find container \"c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9\": container with ID starting with c8f64a2c6b4001fe82a5978f64a1997b1b7a6292b354f817326fc7bcb9dd1dd9 not found: ID does not exist" Nov 03 20:34:43.350623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350595 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.350764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350645 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.350764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350690 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.350886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350854 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.350948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350930 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.351005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.350955 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:43.351065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351030 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.351182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351089 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.351182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351103 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:43.351182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351173 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.351358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351208 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\") pod \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\" (UID: \"5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c\") " Nov 03 20:34:43.351358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351220 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:43.351481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351409 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:43.351703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351677 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:43.351948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351928 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.352000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351955 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.352000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351972 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.352000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.351987 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.352138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.352001 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.353039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.353012 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:43.353197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.353177 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:43.353327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.353298 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "kube-api-access-wjvbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:43.353505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.353489 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token" pod "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" (UID: "5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:43.453474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.453443 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvbp\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-kube-api-access-wjvbp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.453474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.453472 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.453474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.453484 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.453756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.453493 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:43.646471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.646433 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:34:43.651024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.650988 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-wpw7t"] Nov 03 20:34:43.774084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:43.774048 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:44.479504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:44.479466 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.044971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.044937 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:34:45.045217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.045187 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" podUID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" containerName="maas-api" containerID="cri-o://7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1" gracePeriod=30 Nov 03 20:34:45.080522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.080491 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.160736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.160685 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:34:45.161014 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.160960 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" podUID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" containerName="maas-api" containerID="cri-o://03a2f80ff96d52287f9c7548944dc48736532037674c963bd8dedf3a4bf096d1" gracePeriod=30 Nov 03 20:34:45.209223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.209186 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.265842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.265778 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5"] Nov 03 20:34:45.269594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.269561 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-1c6a98-sdxd5"] Nov 03 20:34:45.310669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.310637 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.340479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.340449 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:34:45.343486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.343455 5954 generic.go:356] "Generic (PLEG): container finished" podID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" containerID="03a2f80ff96d52287f9c7548944dc48736532037674c963bd8dedf3a4bf096d1" exitCode=0 Nov 03 20:34:45.343627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.343530 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" event={"ID":"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe","Type":"ContainerDied","Data":"03a2f80ff96d52287f9c7548944dc48736532037674c963bd8dedf3a4bf096d1"} Nov 03 20:34:45.345010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.344986 5954 generic.go:356] "Generic (PLEG): container finished" podID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" containerID="7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1" exitCode=0 Nov 03 20:34:45.345194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.345028 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" event={"ID":"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6","Type":"ContainerDied","Data":"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1"} Nov 03 20:34:45.345194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.345039 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" Nov 03 20:34:45.345194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.345060 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf" event={"ID":"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6","Type":"ContainerDied","Data":"69c9d37a4dd3fe2263bc3b66fecf036d5cae1f9d7255892c238f813c640bd025"} Nov 03 20:34:45.345194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.345081 5954 scope.go:122] "RemoveContainer" containerID="7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1" Nov 03 20:34:45.355661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.355636 5954 scope.go:122] "RemoveContainer" containerID="7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1" Nov 03 20:34:45.356415 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:45.356002 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1\": container with ID starting with 7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1 not found: ID does not exist" containerID="7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1" Nov 03 20:34:45.356415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.356040 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1"} err="failed to get container status \"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1\": rpc error: code = NotFound desc = could not find container \"7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1\": container with ID starting with 7617ce76a0f90c072e793e19b36f9b9204ba31db8caa2cf39ba422e10ca643b1 not found: ID does not exist" Nov 03 20:34:45.453151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.452904 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" path="/var/lib/kubelet/pods/5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c/volumes" Nov 03 20:34:45.458537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.454199 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" path="/var/lib/kubelet/pods/7aa11a05-5aaf-4374-b4f0-6ea56151c1ac/volumes" Nov 03 20:34:45.489574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.488299 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") pod \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " Nov 03 20:34:45.489574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.488382 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\") pod \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " Nov 03 20:34:45.489574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.488441 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\" (UniqueName: \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\") pod \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " Nov 03 20:34:45.489574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.488473 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\") pod \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\" (UID: \"a9718a5d-fbb7-4f03-bb92-f88f7b0715a6\") " Nov 03 20:34:45.491203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.490859 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca" pod "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" (UID: "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:45.506885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.495819 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs" pod "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" (UID: "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:45.506885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.506751 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls" pod "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" (UID: "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:45.507375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.507275 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9" pod "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" (UID: "a9718a5d-fbb7-4f03-bb92-f88f7b0715a6"). InnerVolumeSpecName "kube-api-access-nw8l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:45.524281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.524252 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:34:45.589492 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.589401 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") pod \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " Nov 03 20:34:45.589687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.589500 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\") pod \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " Nov 03 20:34:45.589687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.589562 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\") pod \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " Nov 03 20:34:45.589687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.589634 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\" (UniqueName: \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\") pod \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\" (UID: \"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe\") " Nov 03 20:34:45.589930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.589896 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca" pod "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" (UID: "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:45.590263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.590237 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.590364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.590267 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.590364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.590283 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8l9\" (UniqueName: \"kubernetes.io/projected/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-kube-api-access-nw8l9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.590364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.590299 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.590364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.590311 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.592287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.592259 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls" pod "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" (UID: "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:45.592287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.592280 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs" pod "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" (UID: "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:45.592409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.592297 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn" pod "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" (UID: "0220a2f9-47ea-4a5a-aed5-66a166d2b0fe"). InnerVolumeSpecName "kube-api-access-s7wdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:45.673945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.672685 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:34:45.676792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.676768 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-1c6a98-5f785cf9d9-662hf"] Nov 03 20:34:45.691851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.691826 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.691851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.691852 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wdn\" (UniqueName: \"kubernetes.io/projected/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-kube-api-access-s7wdn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.692030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.691865 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:45.732831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.732798 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.832562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.832522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.956139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:45.955994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.246504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.246350 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.349830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.349791 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.354378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.354354 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" Nov 03 20:34:46.354513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.354373 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l" event={"ID":"0220a2f9-47ea-4a5a-aed5-66a166d2b0fe","Type":"ContainerDied","Data":"b311615d1f74cb06671976c0573303a71d31c60de1095c48e54aa1f657f01b0f"} Nov 03 20:34:46.354513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.354436 5954 scope.go:122] "RemoveContainer" containerID="03a2f80ff96d52287f9c7548944dc48736532037674c963bd8dedf3a4bf096d1" Nov 03 20:34:46.443162 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.443129 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:34:46.447778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.447751 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-62221d-656bc6867-xxn7l"] Nov 03 20:34:46.617254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.617230 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:34:46.708432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.708395 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\") pod \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " Nov 03 20:34:46.708615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.708534 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\" (UniqueName: \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\") pod \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\" (UID: \"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff\") " Nov 03 20:34:46.708795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.708772 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume" pod "5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" (UID: "5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:46.708983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.708964 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:46.710610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.710590 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn" pod "5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" (UID: "5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff"). InnerVolumeSpecName "kube-api-access-9nmxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:46.810085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.809975 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nmxn\" (UniqueName: \"kubernetes.io/projected/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff-kube-api-access-9nmxn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:46.976889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:46.976852 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:47.167264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.167223 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" containerID="cri-o://ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04" gracePeriod=25 Nov 03 20:34:47.180792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.180753 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" containerID="cri-o://5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f" gracePeriod=25 Nov 03 20:34:47.307372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.307332 5954 scope.go:122] "RemoveContainer" containerID="1658dfc4fa260bf3c7db7357279800e6afb6b706ea5e2f738c9afd88dc09b86b" Nov 03 20:34:47.361010 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.360982 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" event={"ID":"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff","Type":"ContainerDied","Data":"1658dfc4fa260bf3c7db7357279800e6afb6b706ea5e2f738c9afd88dc09b86b"} Nov 03 20:34:47.361118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.361042 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" event={"ID":"5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff","Type":"ContainerDied","Data":"ff0fe0da233475791193ace6b8b138ec221627c3dac5ad5c56adf0b3ac8d8d65"} Nov 03 20:34:47.361118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.361070 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6" Nov 03 20:34:47.406073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.406041 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" path="/var/lib/kubelet/pods/0220a2f9-47ea-4a5a-aed5-66a166d2b0fe/volumes" Nov 03 20:34:47.406576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.406560 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" path="/var/lib/kubelet/pods/a9718a5d-fbb7-4f03-bb92-f88f7b0715a6/volumes" Nov 03 20:34:47.406857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.406844 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:34:47.406910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.406874 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-62221d-6cb94d85dc-tfkm6"] Nov 03 20:34:47.796359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:47.796323 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.168359 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:48.168327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.269008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:48.268970 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.661248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:48.661215 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.894998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:48.894959 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:49.204283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204239 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85"] Nov 03 20:34:49.204843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204821 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d356c513-0be7-42ff-88f3-7a3d0a32b581" containerName="revoke-keys" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204844 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d356c513-0be7-42ff-88f3-7a3d0a32b581" containerName="revoke-keys" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204860 5954 cpu_manager.go:414] "containerMap: removing container" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204867 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204878 5954 cpu_manager.go:414] "containerMap: removing container" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204885 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204924 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" containerName="revoke-keys" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204931 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" containerName="revoke-keys" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204952 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204959 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204969 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.204972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.204976 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205014 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205022 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205038 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205045 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205214 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0220a2f9-47ea-4a5a-aed5-66a166d2b0fe" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205229 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9718a5d-fbb7-4f03-bb92-f88f7b0715a6" containerName="maas-api" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205240 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d356c513-0be7-42ff-88f3-7a3d0a32b581" containerName="revoke-keys" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205249 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" containerName="payload-pre-processing" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205261 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7104ef-69c7-4fbf-a2c3-627ef0fcde8c" containerName="istio-proxy" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205276 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa11a05-5aaf-4374-b4f0-6ea56151c1ac" containerName="revoke-keys" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205286 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.205569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.205300 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04bd71-316c-473d-a766-220368ec68f4" containerName="istio-proxy" Nov 03 20:34:49.210732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.210657 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.215316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.215290 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:49.215467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.215403 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:49.217464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.217438 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85"] Nov 03 20:34:49.248353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.248308 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzn8\" (UniqueName: \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.248537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.248374 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.248586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.248560 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.349436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.349397 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzn8\" (UniqueName: \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.349650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.349445 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.349650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.349519 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.350218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.350194 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.351952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.351926 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.383176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.383143 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzn8\" (UniqueName: \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\") pod \"maas-api-revoke-sub-ef7cb93d-zqq85\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.405208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.405175 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff" path="/var/lib/kubelet/pods/5d9c0e0b-7d3d-4950-9175-2e5b2ff7d7ff/volumes" Nov 03 20:34:49.529992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.529907 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:49.567223 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.567192 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:49.811678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:49.811648 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85"] Nov 03 20:34:49.814678 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:34:49.814648 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6da2f3f_0bb1_40db_a0cc_01f667876fa2.slice/crio-de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516 WatchSource:0}: Error finding container de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516: Status 404 returned error can't find the container with id de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516 Nov 03 20:34:50.384639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.384529 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:50.478394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.478352 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" event={"ID":"b6da2f3f-0bb1-40db-a0cc-01f667876fa2","Type":"ContainerStarted","Data":"c5a0a90fa87d9416c162ad4639a5fc6505f2621a4d27812e849c52871ab793c2"} Nov 03 20:34:50.478624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.478422 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" event={"ID":"b6da2f3f-0bb1-40db-a0cc-01f667876fa2","Type":"ContainerStarted","Data":"de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516"} Nov 03 20:34:50.499887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.499830 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" podStartSLOduration=1.499815996 podStartE2EDuration="1.499815996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:50.498871778 +0000 UTC m=+2231.223879385" watchObservedRunningTime="2026-11-03 20:34:50.499815996 +0000 UTC m=+2231.224823589" Nov 03 20:34:50.915849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.915761 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:50.915849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:50.915784 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:51.081346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:51.081307 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:51.486857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:51.486823 5954 generic.go:356] "Generic (PLEG): container finished" podID="b6da2f3f-0bb1-40db-a0cc-01f667876fa2" containerID="c5a0a90fa87d9416c162ad4639a5fc6505f2621a4d27812e849c52871ab793c2" exitCode=0 Nov 03 20:34:51.487312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:51.486870 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" event={"ID":"b6da2f3f-0bb1-40db-a0cc-01f667876fa2","Type":"ContainerDied","Data":"c5a0a90fa87d9416c162ad4639a5fc6505f2621a4d27812e849c52871ab793c2"} Nov 03 20:34:51.794099 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:51.793994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:51.906965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:51.906919 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" containerID="cri-o://e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92" gracePeriod=45 Nov 03 20:34:52.172735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.172700 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:34:52.289476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.289441 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.289672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.289503 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.289672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.289566 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.289672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.289627 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.289985 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290059 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\" (UniqueName: \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290145 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\") pod \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\" (UID: \"193e2f79-fd08-47fc-9c1e-ac4829a83dc9\") " Nov 03 20:34:52.290235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290190 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.290801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290322 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.290801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290715 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.290801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.290737 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.292574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.292548 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.292804 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.292769 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.293061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.293038 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g" pod "193e2f79-fd08-47fc-9c1e-ac4829a83dc9" (UID: "193e2f79-fd08-47fc-9c1e-ac4829a83dc9"). InnerVolumeSpecName "kube-api-access-s956g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.370768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.370728 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:34:52.371036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.371008 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" podUID="d16ae07f-74bc-467b-a290-fea50234a9c6" containerName="maas-api" containerID="cri-o://daf95d62d32e480cc966403b0d5b5da8bdd6cbad22f7252948dc7b163d47e53e" gracePeriod=30 Nov 03 20:34:52.394624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.393094 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s956g\" (UniqueName: \"kubernetes.io/projected/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-kube-api-access-s956g\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.394624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.393150 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.394624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.393165 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.394624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.393179 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/193e2f79-fd08-47fc-9c1e-ac4829a83dc9-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.425129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.425070 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.497233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.497199 5954 generic.go:356] "Generic (PLEG): container finished" podID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerID="e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92" exitCode=0 Nov 03 20:34:52.497670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.497270 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" Nov 03 20:34:52.497670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.497284 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" event={"ID":"193e2f79-fd08-47fc-9c1e-ac4829a83dc9","Type":"ContainerDied","Data":"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92"} Nov 03 20:34:52.497670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.497345 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz" event={"ID":"193e2f79-fd08-47fc-9c1e-ac4829a83dc9","Type":"ContainerDied","Data":"932b73fb39a3d0a319c2379991852a579968051a31d3536fabeb363c621911c0"} Nov 03 20:34:52.497670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.497368 5954 scope.go:122] "RemoveContainer" containerID="e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92" Nov 03 20:34:52.499826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.499504 5954 generic.go:356] "Generic (PLEG): container finished" podID="d16ae07f-74bc-467b-a290-fea50234a9c6" containerID="daf95d62d32e480cc966403b0d5b5da8bdd6cbad22f7252948dc7b163d47e53e" exitCode=0 Nov 03 20:34:52.499826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.499757 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" event={"ID":"d16ae07f-74bc-467b-a290-fea50234a9c6","Type":"ContainerDied","Data":"daf95d62d32e480cc966403b0d5b5da8bdd6cbad22f7252948dc7b163d47e53e"} Nov 03 20:34:52.510045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.510016 5954 scope.go:122] "RemoveContainer" containerID="e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92" Nov 03 20:34:52.510533 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:34:52.510508 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92\": container with ID starting with e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92 not found: ID does not exist" containerID="e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92" Nov 03 20:34:52.510651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.510541 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92"} err="failed to get container status \"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92\": rpc error: code = NotFound desc = could not find container \"e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92\": container with ID starting with e3894fbec5682cb5483ae9a1c33cc2978a8fd970a5ea38e4029a49842b852d92 not found: ID does not exist" Nov 03 20:34:52.523817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.523771 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:34:52.531971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.531925 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-2-w6-d412cf-kserve-57ccf8bffd-v2qmz"] Nov 03 20:34:52.651187 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.604952 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.725789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.725668 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.762177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.762148 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:34:52.799567 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.799530 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\") pod \"d16ae07f-74bc-467b-a290-fea50234a9c6\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " Nov 03 20:34:52.799917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.799598 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\" (UniqueName: \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\") pod \"d16ae07f-74bc-467b-a290-fea50234a9c6\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " Nov 03 20:34:52.799917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.799678 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\") pod \"d16ae07f-74bc-467b-a290-fea50234a9c6\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " Nov 03 20:34:52.799917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.799760 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\") pod \"d16ae07f-74bc-467b-a290-fea50234a9c6\" (UID: \"d16ae07f-74bc-467b-a290-fea50234a9c6\") " Nov 03 20:34:52.800688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.800658 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca" pod "d16ae07f-74bc-467b-a290-fea50234a9c6" (UID: "d16ae07f-74bc-467b-a290-fea50234a9c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:52.802363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.802334 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls" pod "d16ae07f-74bc-467b-a290-fea50234a9c6" (UID: "d16ae07f-74bc-467b-a290-fea50234a9c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.805452 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.805422 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs" pod "d16ae07f-74bc-467b-a290-fea50234a9c6" (UID: "d16ae07f-74bc-467b-a290-fea50234a9c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.806744 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.806723 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8" pod "d16ae07f-74bc-467b-a290-fea50234a9c6" (UID: "d16ae07f-74bc-467b-a290-fea50234a9c6"). InnerVolumeSpecName "kube-api-access-swhv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.823941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.823915 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:52.842692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.842642 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.900906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.900865 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\") pod \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " Nov 03 20:34:52.901171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.900927 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\" (UniqueName: \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\") pod \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " Nov 03 20:34:52.901171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901051 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\") pod \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\" (UID: \"b6da2f3f-0bb1-40db-a0cc-01f667876fa2\") " Nov 03 20:34:52.901378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901323 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca" pod "b6da2f3f-0bb1-40db-a0cc-01f667876fa2" (UID: "b6da2f3f-0bb1-40db-a0cc-01f667876fa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:52.901522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901503 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.901589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901529 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swhv8\" (UniqueName: \"kubernetes.io/projected/d16ae07f-74bc-467b-a290-fea50234a9c6-kube-api-access-swhv8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.901589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901543 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d16ae07f-74bc-467b-a290-fea50234a9c6-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.901589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901556 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d16ae07f-74bc-467b-a290-fea50234a9c6-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.901589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.901568 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.903351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.903325 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca" pod "b6da2f3f-0bb1-40db-a0cc-01f667876fa2" (UID: "b6da2f3f-0bb1-40db-a0cc-01f667876fa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.903578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.903431 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8" pod "b6da2f3f-0bb1-40db-a0cc-01f667876fa2" (UID: "b6da2f3f-0bb1-40db-a0cc-01f667876fa2"). InnerVolumeSpecName "kube-api-access-lzzn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.943793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:52.943644 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.002257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.002222 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzzn8\" (UniqueName: \"kubernetes.io/projected/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-kube-api-access-lzzn8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:53.002257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.002255 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6da2f3f-0bb1-40db-a0cc-01f667876fa2-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:53.136081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.136038 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.260632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.260475 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.405181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.405147 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" path="/var/lib/kubelet/pods/193e2f79-fd08-47fc-9c1e-ac4829a83dc9/volumes" Nov 03 20:34:53.506646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.506612 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" Nov 03 20:34:53.507189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.506643 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr" event={"ID":"d16ae07f-74bc-467b-a290-fea50234a9c6","Type":"ContainerDied","Data":"c54299b3b1ec5853173a5e5244cd221ca903ece8e24f90a7a75e3417ef791a51"} Nov 03 20:34:53.507189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.506703 5954 scope.go:122] "RemoveContainer" containerID="daf95d62d32e480cc966403b0d5b5da8bdd6cbad22f7252948dc7b163d47e53e" Nov 03 20:34:53.509505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.509478 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" event={"ID":"b6da2f3f-0bb1-40db-a0cc-01f667876fa2","Type":"ContainerDied","Data":"de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516"} Nov 03 20:34:53.509505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.509510 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85" Nov 03 20:34:53.509699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.509529 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de699b1c017f509a2530d59c0f8bd6cfb72933bb9122a6bf9940fd465e492516" Nov 03 20:34:53.528916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.528887 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:34:53.534417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.534390 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a744eb-5f48db4684-khpnr"] Nov 03 20:34:53.697441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.697407 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.830659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.830569 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg"] Nov 03 20:34:53.831170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831152 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b6da2f3f-0bb1-40db-a0cc-01f667876fa2" containerName="revoke-keys" Nov 03 20:34:53.831170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831171 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6da2f3f-0bb1-40db-a0cc-01f667876fa2" containerName="revoke-keys" Nov 03 20:34:53.831280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831200 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d16ae07f-74bc-467b-a290-fea50234a9c6" containerName="maas-api" Nov 03 20:34:53.831280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831205 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16ae07f-74bc-467b-a290-fea50234a9c6" containerName="maas-api" Nov 03 20:34:53.831280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831218 5954 cpu_manager.go:414] "containerMap: removing container" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" Nov 03 20:34:53.831280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831223 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" Nov 03 20:34:53.831416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831306 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6da2f3f-0bb1-40db-a0cc-01f667876fa2" containerName="revoke-keys" Nov 03 20:34:53.831416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831320 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16ae07f-74bc-467b-a290-fea50234a9c6" containerName="maas-api" Nov 03 20:34:53.831416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.831327 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="193e2f79-fd08-47fc-9c1e-ac4829a83dc9" containerName="main" Nov 03 20:34:53.835930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.835909 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:53.840272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.840247 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:53.840613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.840587 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:53.844288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.844190 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg"] Nov 03 20:34:53.913340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.913307 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v488l\" (UniqueName: \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:53.913561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.913483 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:53.913561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:53.913557 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.014858 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.014820 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v488l\" (UniqueName: \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.015034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.014956 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.015034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.015027 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.015758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.015737 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.017497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.017476 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.045553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.031568 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v488l\" (UniqueName: \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\") pod \"maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.156604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.156571 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:54.311472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.311431 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:54.429381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.429306 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:54.435714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.435687 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg"] Nov 03 20:34:54.437922 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:34:54.437897 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6c6f31_5afb_41bf_8c85_6f81e4cc8ea2.slice/crio-1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00 WatchSource:0}: Error finding container 1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00: Status 404 returned error can't find the container with id 1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00 Nov 03 20:34:54.517886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.517849 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" event={"ID":"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2","Type":"ContainerStarted","Data":"1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00"} Nov 03 20:34:54.593067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.593032 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:34:54.593339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.593278 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" containerID="cri-o://1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69" gracePeriod=30 Nov 03 20:34:54.651633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:54.651591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:55.045119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.045045 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:55.241680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.241588 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:55.405243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.405200 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16ae07f-74bc-467b-a290-fea50234a9c6" path="/var/lib/kubelet/pods/d16ae07f-74bc-467b-a290-fea50234a9c6/volumes" Nov 03 20:34:55.489462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.489428 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:55.526499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.526393 5954 generic.go:356] "Generic (PLEG): container finished" podID="5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" containerID="149d4185b79aff76239f305baac4a61e27968fdf297f6d4ddd2cb3e9147f3b32" exitCode=0 Nov 03 20:34:55.526499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.526445 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" event={"ID":"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2","Type":"ContainerDied","Data":"149d4185b79aff76239f305baac4a61e27968fdf297f6d4ddd2cb3e9147f3b32"} Nov 03 20:34:55.589583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:55.589531 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:56.273471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.273436 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:56.677043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.677017 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:56.769646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.769615 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\") pod \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " Nov 03 20:34:56.769845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.769769 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\" (UniqueName: \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\") pod \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " Nov 03 20:34:56.769845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.769801 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\") pod \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\" (UID: \"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2\") " Nov 03 20:34:56.769986 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.769962 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca" pod "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" (UID: "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:56.770239 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.770221 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:56.771944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.771918 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca" pod "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" (UID: "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:56.772056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.772045 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l" pod "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" (UID: "5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2"). InnerVolumeSpecName "kube-api-access-v488l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:56.871482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.871443 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v488l\" (UniqueName: \"kubernetes.io/projected/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-kube-api-access-v488l\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:56.871482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.871477 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:56.971757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:56.971714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:57.543995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:57.543964 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" Nov 03 20:34:57.544668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:57.543958 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg" event={"ID":"5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2","Type":"ContainerDied","Data":"1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00"} Nov 03 20:34:57.544668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:57.544661 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e8c3a0171abadecc249c62101123d6b482303c5753fd99b55d69ce39d887a00" Nov 03 20:34:57.567026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:57.566991 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:57.787537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:57.787446 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.039152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.039060 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6"] Nov 03 20:34:58.039607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.039591 5954 cpu_manager.go:414] "containerMap: removing container" podUID="5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" containerName="revoke-keys" Nov 03 20:34:58.039649 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.039609 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" containerName="revoke-keys" Nov 03 20:34:58.039743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.039734 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" containerName="revoke-keys" Nov 03 20:34:58.043078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.043056 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.051393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.051344 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:58.051393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.051366 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:58.053622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.053596 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6"] Nov 03 20:34:58.089305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.089270 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.089305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.089318 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb652\" (UniqueName: \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.089591 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.089527 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.191166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.191092 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.191358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.191248 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.191358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.191282 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb652\" (UniqueName: \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.191794 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.191765 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.193706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.193682 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.204173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.204142 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb652\" (UniqueName: \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\") pod \"maas-api-revoke-sub-b7b76dc6-gbng6\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.358688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.358655 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:34:58.639158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.639126 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6"] Nov 03 20:34:58.641371 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:34:58.641340 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc306b2_c710_4a5d_9185_1b396131a8fa.slice/crio-505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80 WatchSource:0}: Error finding container 505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80: Status 404 returned error can't find the container with id 505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80 Nov 03 20:34:58.650210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.650181 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:34:58.661463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.661437 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.691917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.691885 5954 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:34:58.692078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.691938 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:34:58.757953 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:58.757918 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:59.558092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.558058 5954 generic.go:356] "Generic (PLEG): container finished" podID="3fc306b2-c710-4a5d-9185-1b396131a8fa" containerID="086ce3227e22a672df28483b6c3fdaee7799f6d2107bb8263c0c131fd165c850" exitCode=0 Nov 03 20:34:59.558516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.558161 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" event={"ID":"3fc306b2-c710-4a5d-9185-1b396131a8fa","Type":"ContainerDied","Data":"086ce3227e22a672df28483b6c3fdaee7799f6d2107bb8263c0c131fd165c850"} Nov 03 20:34:59.558516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.558221 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" event={"ID":"3fc306b2-c710-4a5d-9185-1b396131a8fa","Type":"ContainerStarted","Data":"505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80"} Nov 03 20:34:59.589367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.589333 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:59.865969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.865935 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:34:59.916284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916239 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916283 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916306 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916340 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916407 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916458 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916563 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916609 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.916792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916631 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.916792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916735 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.916792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916778 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\") pod \"67f3785c-6715-4731-99f7-3b9f38c90fa3\" (UID: \"67f3785c-6715-4731-99f7-3b9f38c90fa3\") " Nov 03 20:34:59.917063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.916865 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.917140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917064 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:59.917441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917388 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.917441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917423 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f3785c-6715-4731-99f7-3b9f38c90fa3-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.917441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917444 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.917663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917453 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.917663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.917465 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.919460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.919435 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:59.919585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.919556 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "kube-api-access-tl6ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:59.919699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.919662 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:59.920653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:34:59.920635 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo" pod "67f3785c-6715-4731-99f7-3b9f38c90fa3" (UID: "67f3785c-6715-4731-99f7-3b9f38c90fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:00.018682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.018648 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.018682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.018678 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.018682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.018689 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.018944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.018706 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6ws\" (UniqueName: \"kubernetes.io/projected/67f3785c-6715-4731-99f7-3b9f38c90fa3-kube-api-access-tl6ws\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.018944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.018718 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f3785c-6715-4731-99f7-3b9f38c90fa3-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.175366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.175325 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.275978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.275671 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.279913 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:00.279882 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:35:00.373962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.373713 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.502413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.502320 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:35:00.502946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.502912 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" containerID="cri-o://ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14" gracePeriod=30 Nov 03 20:35:00.538586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.538552 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.566808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.566770 5954 generic.go:356] "Generic (PLEG): container finished" podID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerID="1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69" exitCode=0 Nov 03 20:35:00.567280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.566818 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" Nov 03 20:35:00.567280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.566808 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerDied","Data":"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69"} Nov 03 20:35:00.567280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.566943 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd" event={"ID":"67f3785c-6715-4731-99f7-3b9f38c90fa3","Type":"ContainerDied","Data":"a977daf411e0c093bbdd7be84f97edf697784f0c8878120ff26681db20cdffe1"} Nov 03 20:35:00.567280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.566965 5954 scope.go:122] "RemoveContainer" containerID="1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69" Nov 03 20:35:00.578277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.578260 5954 scope.go:122] "RemoveContainer" containerID="97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42" Nov 03 20:35:00.605127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.590004 5954 scope.go:122] "RemoveContainer" containerID="1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69" Nov 03 20:35:00.605127 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:00.595450 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69\": container with ID starting with 1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69 not found: ID does not exist" containerID="1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69" Nov 03 20:35:00.605127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.595509 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69"} err="failed to get container status \"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69\": rpc error: code = NotFound desc = could not find container \"1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69\": container with ID starting with 1e61ba890a1b1ad069c2bb9bb261287c6bc982f3e7d5f2418e369c0abf5eff69 not found: ID does not exist" Nov 03 20:35:00.605127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.595534 5954 scope.go:122] "RemoveContainer" containerID="97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42" Nov 03 20:35:00.629609 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:00.629561 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42\": container with ID starting with 97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42 not found: ID does not exist" containerID="97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42" Nov 03 20:35:00.629830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.629618 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42"} err="failed to get container status \"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42\": rpc error: code = NotFound desc = could not find container \"97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42\": container with ID starting with 97c7accb7f3529bd77703b8ff51d92ce8bbb7ab01b51fb58e97c4ac032ad9a42 not found: ID does not exist" Nov 03 20:35:00.636202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.636155 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:35:00.637035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.636981 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-g6fzd"] Nov 03 20:35:00.783912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.783886 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:35:00.834136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.834089 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\") pod \"3fc306b2-c710-4a5d-9185-1b396131a8fa\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " Nov 03 20:35:00.834290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.834169 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\") pod \"3fc306b2-c710-4a5d-9185-1b396131a8fa\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " Nov 03 20:35:00.834290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.834229 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\" (UniqueName: \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\") pod \"3fc306b2-c710-4a5d-9185-1b396131a8fa\" (UID: \"3fc306b2-c710-4a5d-9185-1b396131a8fa\") " Nov 03 20:35:00.834596 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.834566 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca" pod "3fc306b2-c710-4a5d-9185-1b396131a8fa" (UID: "3fc306b2-c710-4a5d-9185-1b396131a8fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:00.834729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.834693 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.835534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.835508 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:35:00.836451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.836422 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652" pod "3fc306b2-c710-4a5d-9185-1b396131a8fa" (UID: "3fc306b2-c710-4a5d-9185-1b396131a8fa"). InnerVolumeSpecName "kube-api-access-vb652". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.836568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.836524 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca" pod "3fc306b2-c710-4a5d-9185-1b396131a8fa" (UID: "3fc306b2-c710-4a5d-9185-1b396131a8fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:00.916385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.916338 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:00.916385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.916360 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:00.935417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.935384 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb652\" (UniqueName: \"kubernetes.io/projected/3fc306b2-c710-4a5d-9185-1b396131a8fa-kube-api-access-vb652\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.935417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:00.935416 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fc306b2-c710-4a5d-9185-1b396131a8fa-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:01.015416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.015317 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:01.209414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209378 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z"] Nov 03 20:35:01.209925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209910 5954 cpu_manager.go:414] "containerMap: removing container" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.209996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209926 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.209996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209942 5954 cpu_manager.go:414] "containerMap: removing container" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.209996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209950 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.209996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209991 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3fc306b2-c710-4a5d-9185-1b396131a8fa" containerName="revoke-keys" Nov 03 20:35:01.210171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.209998 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc306b2-c710-4a5d-9185-1b396131a8fa" containerName="revoke-keys" Nov 03 20:35:01.210171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.210077 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc306b2-c710-4a5d-9185-1b396131a8fa" containerName="revoke-keys" Nov 03 20:35:01.210171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.210089 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.210171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.210099 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" containerName="istio-proxy" Nov 03 20:35:01.213694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.213677 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.225075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.222827 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z"] Nov 03 20:35:01.238030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.237977 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.238256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.238161 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78pd\" (UniqueName: \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.238319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.238263 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.238730 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.238695 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:01.333362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.333275 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:01.339760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.339723 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.339854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.339809 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78pd\" (UniqueName: \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.339903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.339889 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.340456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.340437 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.342366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.342341 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.354879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.354856 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78pd\" (UniqueName: \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\") pod \"maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.404317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.404285 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f3785c-6715-4731-99f7-3b9f38c90fa3" path="/var/lib/kubelet/pods/67f3785c-6715-4731-99f7-3b9f38c90fa3/volumes" Nov 03 20:35:01.534415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.534379 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:01.575919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.575881 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" event={"ID":"3fc306b2-c710-4a5d-9185-1b396131a8fa","Type":"ContainerDied","Data":"505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80"} Nov 03 20:35:01.575919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.575912 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6" Nov 03 20:35:01.576369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.575946 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505420caa7c5c10a9d44a85d9c8cc6383c87cb15c7282a160f9e5121cf5c6e80" Nov 03 20:35:01.819824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:01.819786 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z"] Nov 03 20:35:01.822244 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:01.822217 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37c7f13_9fd9_4632_8d04_0221d9c0f05c.slice/crio-f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d WatchSource:0}: Error finding container f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d: Status 404 returned error can't find the container with id f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d Nov 03 20:35:02.228012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:02.227967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:02.584097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:02.583999 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" event={"ID":"d37c7f13-9fd9-4632-8d04-0221d9c0f05c","Type":"ContainerStarted","Data":"06be7614a18263fcc4301a0d1b6fc96a15eef26066c86edd4837ac3e29b56a7e"} Nov 03 20:35:02.584097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:02.584053 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" event={"ID":"d37c7f13-9fd9-4632-8d04-0221d9c0f05c","Type":"ContainerStarted","Data":"f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d"} Nov 03 20:35:02.608087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:02.608034 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" podStartSLOduration=1.608019687 podStartE2EDuration="1.608019687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:02.607426926 +0000 UTC m=+2243.332434557" watchObservedRunningTime="2026-11-03 20:35:02.608019687 +0000 UTC m=+2243.333027305" Nov 03 20:35:03.115806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:03.115773 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:03.524013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:03.523927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:03.592130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:03.592074 5954 generic.go:356] "Generic (PLEG): container finished" podID="d37c7f13-9fd9-4632-8d04-0221d9c0f05c" containerID="06be7614a18263fcc4301a0d1b6fc96a15eef26066c86edd4837ac3e29b56a7e" exitCode=0 Nov 03 20:35:03.592530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:03.592175 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" event={"ID":"d37c7f13-9fd9-4632-8d04-0221d9c0f05c","Type":"ContainerDied","Data":"06be7614a18263fcc4301a0d1b6fc96a15eef26066c86edd4837ac3e29b56a7e"} Nov 03 20:35:03.991810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:03.991759 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.106747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.106712 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.207203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.207172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.744880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.744854 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:04.779680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.779642 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\") pod \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " Nov 03 20:35:04.779867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.779752 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\") pod \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " Nov 03 20:35:04.779947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.779862 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\" (UniqueName: \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\") pod \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\" (UID: \"d37c7f13-9fd9-4632-8d04-0221d9c0f05c\") " Nov 03 20:35:04.780118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.780077 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca" pod "d37c7f13-9fd9-4632-8d04-0221d9c0f05c" (UID: "d37c7f13-9fd9-4632-8d04-0221d9c0f05c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:04.780394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.780366 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:04.781945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.781907 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca" pod "d37c7f13-9fd9-4632-8d04-0221d9c0f05c" (UID: "d37c7f13-9fd9-4632-8d04-0221d9c0f05c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:04.782071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.782038 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd" pod "d37c7f13-9fd9-4632-8d04-0221d9c0f05c" (UID: "d37c7f13-9fd9-4632-8d04-0221d9c0f05c"). InnerVolumeSpecName "kube-api-access-r78pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:04.798219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.798191 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.881769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.881735 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:04.881769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:04.881766 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78pd\" (UniqueName: \"kubernetes.io/projected/d37c7f13-9fd9-4632-8d04-0221d9c0f05c-kube-api-access-r78pd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.486261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.486224 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.605095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.605067 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" Nov 03 20:35:05.605298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.605133 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z" event={"ID":"d37c7f13-9fd9-4632-8d04-0221d9c0f05c","Type":"ContainerDied","Data":"f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d"} Nov 03 20:35:05.605298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.605181 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f72b1453cd48196734b59506135737cc87b089bf45ddc49f44553666c310497d" Nov 03 20:35:05.673557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.671270 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.809437 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.809399 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb"] Nov 03 20:35:05.810281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.810243 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.814003 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.813976 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-d412cf-n58nb"] Nov 03 20:35:05.846261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.846231 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:35:05.895732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895681 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.895918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895743 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.895918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895822 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.895918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895871 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.895918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895910 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.896188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.895962 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.896188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896009 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.896314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896196 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.896314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896197 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:05.896314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896290 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\") pod \"eaf20803-6403-415c-b166-27058ff3e898\" (UID: \"eaf20803-6403-415c-b166-27058ff3e898\") " Nov 03 20:35:05.896472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896445 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.896541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896463 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.896765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896741 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.896884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896832 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.897017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.896992 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.897017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.897017 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.897242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.897032 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaf20803-6403-415c-b166-27058ff3e898-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.897242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.897048 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.897242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.897062 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.898463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.898434 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.898911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.898882 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "kube-api-access-8pf2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:05.898911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.898899 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:05.899047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.898917 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo" pod "eaf20803-6403-415c-b166-27058ff3e898" (UID: "eaf20803-6403-415c-b166-27058ff3e898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:05.977319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.977207 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.998548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.998511 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pf2g\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-kube-api-access-8pf2g\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.998548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.998545 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaf20803-6403-415c-b166-27058ff3e898-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.998548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.998556 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaf20803-6403-415c-b166-27058ff3e898-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:05.998800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:05.998566 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaf20803-6403-415c-b166-27058ff3e898-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.083525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.083496 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:06.612652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.612615 5954 generic.go:356] "Generic (PLEG): container finished" podID="eaf20803-6403-415c-b166-27058ff3e898" containerID="ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14" exitCode=0 Nov 03 20:35:06.612652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.612658 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" event={"ID":"eaf20803-6403-415c-b166-27058ff3e898","Type":"ContainerDied","Data":"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14"} Nov 03 20:35:06.612938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.612670 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" Nov 03 20:35:06.612938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.612703 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb" event={"ID":"eaf20803-6403-415c-b166-27058ff3e898","Type":"ContainerDied","Data":"d747e012ec159fe210c105a463f531a0e62bc9eb34722b6ae7031ed7358f75f6"} Nov 03 20:35:06.612938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.612732 5954 scope.go:122] "RemoveContainer" containerID="ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14" Nov 03 20:35:06.623901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.623707 5954 scope.go:122] "RemoveContainer" containerID="ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14" Nov 03 20:35:06.623998 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:06.623980 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14\": container with ID starting with ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14 not found: ID does not exist" containerID="ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14" Nov 03 20:35:06.624047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.624007 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14"} err="failed to get container status \"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14\": rpc error: code = NotFound desc = could not find container \"ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14\": container with ID starting with ca2730d5b4a9f944015c1e5acdbef7758343e923bb6c63108f81089d68cd7e14 not found: ID does not exist" Nov 03 20:35:06.641594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.641549 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:35:06.645715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.645684 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-94zzb"] Nov 03 20:35:06.995381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:06.995279 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.406176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.406142 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d356c513-0be7-42ff-88f3-7a3d0a32b581" path="/var/lib/kubelet/pods/d356c513-0be7-42ff-88f3-7a3d0a32b581/volumes" Nov 03 20:35:07.406737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.406714 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf20803-6403-415c-b166-27058ff3e898" path="/var/lib/kubelet/pods/eaf20803-6403-415c-b166-27058ff3e898/volumes" Nov 03 20:35:07.452795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.452764 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:35:07.495845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.495781 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:35:07.516571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.516533 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\" (UniqueName: \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\") pod \"addd9bf4-22f0-400b-957a-7a090243bdd0\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " Nov 03 20:35:07.516830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.516781 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\" (UniqueName: \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\") pod \"aaa553fb-c462-4556-8684-4e78da36aa00\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " Nov 03 20:35:07.516937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.516913 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\") pod \"aaa553fb-c462-4556-8684-4e78da36aa00\" (UID: \"aaa553fb-c462-4556-8684-4e78da36aa00\") " Nov 03 20:35:07.517077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.516954 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\") pod \"addd9bf4-22f0-400b-957a-7a090243bdd0\" (UID: \"addd9bf4-22f0-400b-957a-7a090243bdd0\") " Nov 03 20:35:07.517451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.517418 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume" pod "aaa553fb-c462-4556-8684-4e78da36aa00" (UID: "aaa553fb-c462-4556-8684-4e78da36aa00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:07.517762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.517733 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume" pod "addd9bf4-22f0-400b-957a-7a090243bdd0" (UID: "addd9bf4-22f0-400b-957a-7a090243bdd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:07.520083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.520051 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl" pod "aaa553fb-c462-4556-8684-4e78da36aa00" (UID: "aaa553fb-c462-4556-8684-4e78da36aa00"). InnerVolumeSpecName "kube-api-access-6hkhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:07.520325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.520302 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc" pod "addd9bf4-22f0-400b-957a-7a090243bdd0" (UID: "addd9bf4-22f0-400b-957a-7a090243bdd0"). InnerVolumeSpecName "kube-api-access-krlrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:07.548147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548088 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx"] Nov 03 20:35:07.548812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548786 5954 cpu_manager.go:414] "containerMap: removing container" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" Nov 03 20:35:07.548812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548810 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548824 5954 cpu_manager.go:414] "containerMap: removing container" podUID="d37c7f13-9fd9-4632-8d04-0221d9c0f05c" containerName="revoke-keys" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548829 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37c7f13-9fd9-4632-8d04-0221d9c0f05c" containerName="revoke-keys" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548840 5954 cpu_manager.go:414] "containerMap: removing container" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548846 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548862 5954 cpu_manager.go:414] "containerMap: removing container" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" Nov 03 20:35:07.548961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548867 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" Nov 03 20:35:07.549209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548965 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" containerName="payload-pre-processing" Nov 03 20:35:07.549209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548973 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37c7f13-9fd9-4632-8d04-0221d9c0f05c" containerName="revoke-keys" Nov 03 20:35:07.549209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548979 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerName="payload-processing" Nov 03 20:35:07.549209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.548986 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf20803-6403-415c-b166-27058ff3e898" containerName="istio-proxy" Nov 03 20:35:07.552414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.552394 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.563828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.563802 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:07.564355 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.564079 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:35:07.570188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.570161 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx"] Nov 03 20:35:07.570561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.570540 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9"] Nov 03 20:35:07.574584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.574560 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.583185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.583158 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9"] Nov 03 20:35:07.618486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618453 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rnww\" (UniqueName: \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.618666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618525 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.618666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618646 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.618771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618755 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99w7d\" (UniqueName: \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.618831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618817 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.618869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.618848 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.619026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.619012 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/addd9bf4-22f0-400b-957a-7a090243bdd0-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:07.619063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.619029 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlrc\" (UniqueName: \"kubernetes.io/projected/addd9bf4-22f0-400b-957a-7a090243bdd0-kube-api-access-krlrc\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:07.619063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.619040 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkhl\" (UniqueName: \"kubernetes.io/projected/aaa553fb-c462-4556-8684-4e78da36aa00-kube-api-access-6hkhl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:07.619063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.619051 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaa553fb-c462-4556-8684-4e78da36aa00-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:07.621193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.621170 5954 generic.go:356] "Generic (PLEG): container finished" podID="addd9bf4-22f0-400b-957a-7a090243bdd0" containerID="ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04" exitCode=0 Nov 03 20:35:07.621320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.621223 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" Nov 03 20:35:07.621320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.621263 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" event={"ID":"addd9bf4-22f0-400b-957a-7a090243bdd0","Type":"ContainerDied","Data":"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04"} Nov 03 20:35:07.621447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.621322 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f" event={"ID":"addd9bf4-22f0-400b-957a-7a090243bdd0","Type":"ContainerDied","Data":"3ac0f319a81495a23051ca36a6ff00f772f4fa66816237ce18edc61195200f1f"} Nov 03 20:35:07.621447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.621354 5954 scope.go:122] "RemoveContainer" containerID="ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04" Nov 03 20:35:07.622844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.622812 5954 generic.go:356] "Generic (PLEG): container finished" podID="aaa553fb-c462-4556-8684-4e78da36aa00" containerID="5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f" exitCode=0 Nov 03 20:35:07.622958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.622839 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" event={"ID":"aaa553fb-c462-4556-8684-4e78da36aa00","Type":"ContainerDied","Data":"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f"} Nov 03 20:35:07.622958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.622877 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" Nov 03 20:35:07.622958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.622890 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8" event={"ID":"aaa553fb-c462-4556-8684-4e78da36aa00","Type":"ContainerDied","Data":"b2792fa3add155d711885506690a58e742333653c0fa17fe00126c16869e6d5d"} Nov 03 20:35:07.633076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.632955 5954 scope.go:122] "RemoveContainer" containerID="ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04" Nov 03 20:35:07.633262 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:07.633242 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04\": container with ID starting with ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04 not found: ID does not exist" containerID="ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04" Nov 03 20:35:07.633349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.633274 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04"} err="failed to get container status \"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04\": rpc error: code = NotFound desc = could not find container \"ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04\": container with ID starting with ddd0ace06718cdab255e54020f39f750e21591b87ae8afec56641aefa4691c04 not found: ID does not exist" Nov 03 20:35:07.633349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.633296 5954 scope.go:122] "RemoveContainer" containerID="5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f" Nov 03 20:35:07.641701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.641683 5954 scope.go:122] "RemoveContainer" containerID="5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f" Nov 03 20:35:07.641947 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:07.641932 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f\": container with ID starting with 5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f not found: ID does not exist" containerID="5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f" Nov 03 20:35:07.642001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.641955 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f"} err="failed to get container status \"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f\": rpc error: code = NotFound desc = could not find container \"5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f\": container with ID starting with 5af80049d300064ccf438c5e214c3f14097013e85269b19b8671cf5c9e76f37f not found: ID does not exist" Nov 03 20:35:07.645594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.645571 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:35:07.649746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.649725 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-d412cf-849b9c959d-ccm2f"] Nov 03 20:35:07.660456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.660432 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:35:07.664471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.664449 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d412cf-768746c6dc-6sdf8"] Nov 03 20:35:07.720205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720166 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99w7d\" (UniqueName: \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.720397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720254 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.720397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720284 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.720397 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720372 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rnww\" (UniqueName: \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.720560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720443 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.720635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.720611 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.721100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.721078 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.721293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.721276 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.722750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.722720 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.722750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.722743 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.732945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.732919 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99w7d\" (UniqueName: \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\") pod \"maas-api-revoke-sub-bc4adfaf-qs7g9\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:07.733067 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.733027 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rnww\" (UniqueName: \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\") pod \"maas-api-revoke-sub-5aae205c-l6rgx\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.866886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.866845 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:07.876919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.876873 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.894228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:07.894203 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:08.183962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.183930 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx"] Nov 03 20:35:08.186045 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:08.186013 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9fc097_cec8_4a6b_9fe6_0841ef43c582.slice/crio-4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63 WatchSource:0}: Error finding container 4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63: Status 404 returned error can't find the container with id 4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63 Nov 03 20:35:08.217000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.216964 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9"] Nov 03 20:35:08.218659 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:08.218630 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f682e1_e5e5_402b_a568_c7bed86c6ee2.slice/crio-c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2 WatchSource:0}: Error finding container c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2: Status 404 returned error can't find the container with id c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2 Nov 03 20:35:08.601573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.601538 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:08.633656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.633622 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" event={"ID":"f3f682e1-e5e5-402b-a568-c7bed86c6ee2","Type":"ContainerStarted","Data":"7432c22e6d122ee63bdfe497e3a976fea9d73079bd5bc6dbd6fe43be98baf671"} Nov 03 20:35:08.633799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.633671 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" event={"ID":"f3f682e1-e5e5-402b-a568-c7bed86c6ee2","Type":"ContainerStarted","Data":"c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2"} Nov 03 20:35:08.635292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.635269 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" event={"ID":"7a9fc097-cec8-4a6b-9fe6-0841ef43c582","Type":"ContainerStarted","Data":"3caf5e13f922ee31f9711c790522cd959343fc374acd7cbd75597b50894079d0"} Nov 03 20:35:08.635424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.635306 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" event={"ID":"7a9fc097-cec8-4a6b-9fe6-0841ef43c582","Type":"ContainerStarted","Data":"4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63"} Nov 03 20:35:08.668957 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.665257 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" podStartSLOduration=1.665235296 podStartE2EDuration="1.665235296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:08.659371595 +0000 UTC m=+2249.384379193" watchObservedRunningTime="2026-11-03 20:35:08.665235296 +0000 UTC m=+2249.390242895" Nov 03 20:35:08.683444 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:08.683375 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" podStartSLOduration=1.683340337 podStartE2EDuration="1.683340337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:08.678164919 +0000 UTC m=+2249.403172513" watchObservedRunningTime="2026-11-03 20:35:08.683340337 +0000 UTC m=+2249.408347936" Nov 03 20:35:09.297874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.297785 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:09.405090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.405055 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa553fb-c462-4556-8684-4e78da36aa00" path="/var/lib/kubelet/pods/aaa553fb-c462-4556-8684-4e78da36aa00/volumes" Nov 03 20:35:09.405496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.405481 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addd9bf4-22f0-400b-957a-7a090243bdd0" path="/var/lib/kubelet/pods/addd9bf4-22f0-400b-957a-7a090243bdd0/volumes" Nov 03 20:35:09.643461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.643426 5954 generic.go:356] "Generic (PLEG): container finished" podID="f3f682e1-e5e5-402b-a568-c7bed86c6ee2" containerID="7432c22e6d122ee63bdfe497e3a976fea9d73079bd5bc6dbd6fe43be98baf671" exitCode=0 Nov 03 20:35:09.643646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.643518 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" event={"ID":"f3f682e1-e5e5-402b-a568-c7bed86c6ee2","Type":"ContainerDied","Data":"7432c22e6d122ee63bdfe497e3a976fea9d73079bd5bc6dbd6fe43be98baf671"} Nov 03 20:35:09.644997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.644974 5954 generic.go:356] "Generic (PLEG): container finished" podID="7a9fc097-cec8-4a6b-9fe6-0841ef43c582" containerID="3caf5e13f922ee31f9711c790522cd959343fc374acd7cbd75597b50894079d0" exitCode=0 Nov 03 20:35:09.645138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:09.645052 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" event={"ID":"7a9fc097-cec8-4a6b-9fe6-0841ef43c582","Type":"ContainerDied","Data":"3caf5e13f922ee31f9711c790522cd959343fc374acd7cbd75597b50894079d0"} Nov 03 20:35:10.067306 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.067209 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:10.773102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.773034 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:10.850068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.850037 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:10.932427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.932389 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961382 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\" (UniqueName: \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\") pod \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961511 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\") pod \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961553 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\") pod \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961585 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\") pod \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961638 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\" (UniqueName: \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\") pod \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\" (UID: \"f3f682e1-e5e5-402b-a568-c7bed86c6ee2\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.961758 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\") pod \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\" (UID: \"7a9fc097-cec8-4a6b-9fe6-0841ef43c582\") " Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.963097 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca" pod "7a9fc097-cec8-4a6b-9fe6-0841ef43c582" (UID: "7a9fc097-cec8-4a6b-9fe6-0841ef43c582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:10.964676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.963608 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca" pod "f3f682e1-e5e5-402b-a568-c7bed86c6ee2" (UID: "f3f682e1-e5e5-402b-a568-c7bed86c6ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:10.965915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.965769 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca" pod "f3f682e1-e5e5-402b-a568-c7bed86c6ee2" (UID: "f3f682e1-e5e5-402b-a568-c7bed86c6ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:10.965915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.966673 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca" pod "7a9fc097-cec8-4a6b-9fe6-0841ef43c582" (UID: "7a9fc097-cec8-4a6b-9fe6-0841ef43c582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:10.965915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.967772 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d" pod "f3f682e1-e5e5-402b-a568-c7bed86c6ee2" (UID: "f3f682e1-e5e5-402b-a568-c7bed86c6ee2"). InnerVolumeSpecName "kube-api-access-99w7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:10.969013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:10.968975 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww" pod "7a9fc097-cec8-4a6b-9fe6-0841ef43c582" (UID: "7a9fc097-cec8-4a6b-9fe6-0841ef43c582"). InnerVolumeSpecName "kube-api-access-8rnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:11.063295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063244 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.063295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063293 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.063524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063311 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.063524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063326 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99w7d\" (UniqueName: \"kubernetes.io/projected/f3f682e1-e5e5-402b-a568-c7bed86c6ee2-kube-api-access-99w7d\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.063524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063341 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.063524 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.063356 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rnww\" (UniqueName: \"kubernetes.io/projected/7a9fc097-cec8-4a6b-9fe6-0841ef43c582-kube-api-access-8rnww\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:11.326076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.325990 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:35:11.402509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.402481 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.502428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.502388 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.604219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.604183 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.676464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.676425 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" event={"ID":"f3f682e1-e5e5-402b-a568-c7bed86c6ee2","Type":"ContainerDied","Data":"c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2"} Nov 03 20:35:11.676655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.676493 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e2d596a924e3e82c580861c92fa68a4abeb09cc2c630f9c4136b502f1a18b2" Nov 03 20:35:11.676655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.676563 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9" Nov 03 20:35:11.680361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.679613 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" event={"ID":"7a9fc097-cec8-4a6b-9fe6-0841ef43c582","Type":"ContainerDied","Data":"4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63"} Nov 03 20:35:11.680361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.679684 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc4a72d5c9fdee2c38039c418eb822bde5a04972e4b76a542e565b670c38b63" Nov 03 20:35:11.680361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:11.679791 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx" Nov 03 20:35:12.123441 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123404 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr"] Nov 03 20:35:12.123906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123893 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7a9fc097-cec8-4a6b-9fe6-0841ef43c582" containerName="revoke-keys" Nov 03 20:35:12.123969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123907 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9fc097-cec8-4a6b-9fe6-0841ef43c582" containerName="revoke-keys" Nov 03 20:35:12.123969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123919 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f3f682e1-e5e5-402b-a568-c7bed86c6ee2" containerName="revoke-keys" Nov 03 20:35:12.123969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123926 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f682e1-e5e5-402b-a568-c7bed86c6ee2" containerName="revoke-keys" Nov 03 20:35:12.124063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.123998 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9fc097-cec8-4a6b-9fe6-0841ef43c582" containerName="revoke-keys" Nov 03 20:35:12.124063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.124007 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f682e1-e5e5-402b-a568-c7bed86c6ee2" containerName="revoke-keys" Nov 03 20:35:12.127514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.127493 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.135070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.135034 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:35:12.135177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.135074 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:12.138386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.138350 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt"] Nov 03 20:35:12.144173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.144151 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr"] Nov 03 20:35:12.144313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.144271 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.159780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.159747 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt"] Nov 03 20:35:12.178097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178060 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7n4\" (UniqueName: \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.178298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178122 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.178342 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178314 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.178433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178414 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.178472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178453 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.178559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.178547 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rp4q\" (UniqueName: \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.279635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279600 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.279872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279657 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.279872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279690 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.279872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279736 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rp4q\" (UniqueName: \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.279872 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279762 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7n4\" (UniqueName: \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.280150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.279881 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.280408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.280385 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.280482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.280402 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.282185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.282161 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.282309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.282288 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.292070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.292033 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rp4q\" (UniqueName: \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\") pod \"maas-api-revoke-sub-e908b41b-bsmqt\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.292197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.292130 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7n4\" (UniqueName: \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\") pod \"maas-api-revoke-sub-12b4a4d7-snkpr\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.444233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.444096 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:12.450776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.450743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:12.464512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.464483 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:12.746562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.746535 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr"] Nov 03 20:35:12.747554 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:12.747530 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295784c0_89db_405f_b53d_3533f77b8098.slice/crio-2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf WatchSource:0}: Error finding container 2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf: Status 404 returned error can't find the container with id 2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf Nov 03 20:35:12.791848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:12.791802 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt"] Nov 03 20:35:12.800170 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:12.800133 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13687593_3523_4ed6_8770_43766068baa6.slice/crio-c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063 WatchSource:0}: Error finding container c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063: Status 404 returned error can't find the container with id c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063 Nov 03 20:35:13.164893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.164857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.273841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.273732 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.387033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.386997 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.622427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.622391 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.675368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.675316 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" containerID="cri-o://4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41" gracePeriod=45 Nov 03 20:35:13.731138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.731001 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused" Nov 03 20:35:13.773266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.761362 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" event={"ID":"13687593-3523-4ed6-8770-43766068baa6","Type":"ContainerStarted","Data":"d637878c68fb123f6942751d5891b6b99f52fb518bdbd364095eafc41126d7f4"} Nov 03 20:35:13.773266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.761440 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" event={"ID":"13687593-3523-4ed6-8770-43766068baa6","Type":"ContainerStarted","Data":"c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063"} Nov 03 20:35:13.773266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.763994 5954 generic.go:356] "Generic (PLEG): container finished" podID="295784c0-89db-405f-b53d-3533f77b8098" containerID="96811fb867558330bdc81eef3f7b982a318fcb9cc21e4f15ae1e8311c94bdae4" exitCode=0 Nov 03 20:35:13.773266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.764038 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" event={"ID":"295784c0-89db-405f-b53d-3533f77b8098","Type":"ContainerDied","Data":"96811fb867558330bdc81eef3f7b982a318fcb9cc21e4f15ae1e8311c94bdae4"} Nov 03 20:35:13.773266 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.764073 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" event={"ID":"295784c0-89db-405f-b53d-3533f77b8098","Type":"ContainerStarted","Data":"2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf"} Nov 03 20:35:13.842813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.842757 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" podStartSLOduration=1.842737725 podStartE2EDuration="1.842737725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:13.840404255 +0000 UTC m=+2254.565411853" watchObservedRunningTime="2026-11-03 20:35:13.842737725 +0000 UTC m=+2254.567745322" Nov 03 20:35:13.924746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:13.923092 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:14.001654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.001626 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:35:14.048726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048684 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.048909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048766 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\" (UniqueName: \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.048909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048805 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.048909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048873 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.049125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048925 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.049125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.048954 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\") pod \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\" (UID: \"9efdcc23-c3fc-447b-92cb-a9decf3ece60\") " Nov 03 20:35:14.049125 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049013 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.049367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049341 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.049439 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049381 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.049579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049553 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.049579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049580 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.049761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.049594 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.052466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.052444 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "kube-api-access-fsps9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:14.052466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.052447 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:14.059563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.059533 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm" pod "9efdcc23-c3fc-447b-92cb-a9decf3ece60" (UID: "9efdcc23-c3fc-447b-92cb-a9decf3ece60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.151192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.150776 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efdcc23-c3fc-447b-92cb-a9decf3ece60-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.151192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.150812 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsps9\" (UniqueName: \"kubernetes.io/projected/9efdcc23-c3fc-447b-92cb-a9decf3ece60-kube-api-access-fsps9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.151192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.150830 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9efdcc23-c3fc-447b-92cb-a9decf3ece60-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.243415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.243377 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:14.772603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.772568 5954 generic.go:356] "Generic (PLEG): container finished" podID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerID="4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41" exitCode=0 Nov 03 20:35:14.772830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.772613 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" Nov 03 20:35:14.772830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.772648 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" event={"ID":"9efdcc23-c3fc-447b-92cb-a9decf3ece60","Type":"ContainerDied","Data":"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41"} Nov 03 20:35:14.772830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.772683 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf" event={"ID":"9efdcc23-c3fc-447b-92cb-a9decf3ece60","Type":"ContainerDied","Data":"c9867f804a43cc0ca3ac61702acd80d193211a9c8e3a1cfe1e6ca45b1134ba42"} Nov 03 20:35:14.772830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.772704 5954 scope.go:122] "RemoveContainer" containerID="4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41" Nov 03 20:35:14.774581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.774558 5954 generic.go:356] "Generic (PLEG): container finished" podID="13687593-3523-4ed6-8770-43766068baa6" containerID="d637878c68fb123f6942751d5891b6b99f52fb518bdbd364095eafc41126d7f4" exitCode=0 Nov 03 20:35:14.774709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.774649 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" event={"ID":"13687593-3523-4ed6-8770-43766068baa6","Type":"ContainerDied","Data":"d637878c68fb123f6942751d5891b6b99f52fb518bdbd364095eafc41126d7f4"} Nov 03 20:35:14.784337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.784305 5954 scope.go:122] "RemoveContainer" containerID="4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41" Nov 03 20:35:14.784637 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:14.784619 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41\": container with ID starting with 4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41 not found: ID does not exist" containerID="4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41" Nov 03 20:35:14.784701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.784646 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41"} err="failed to get container status \"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41\": rpc error: code = NotFound desc = could not find container \"4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41\": container with ID starting with 4dd352ee8ab0f521b85f619b32e32aa6843129f6cc77051be9b8428c0b262e41 not found: ID does not exist" Nov 03 20:35:14.815138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.815077 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:35:14.820044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.820012 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/distinct-2-w4-3aba31-kserve-7fc97fc595-8z7sf"] Nov 03 20:35:14.934020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:14.933993 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:15.070852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.070764 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\") pod \"295784c0-89db-405f-b53d-3533f77b8098\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " Nov 03 20:35:15.071021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.070931 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\") pod \"295784c0-89db-405f-b53d-3533f77b8098\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " Nov 03 20:35:15.071021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.070987 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\" (UniqueName: \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\") pod \"295784c0-89db-405f-b53d-3533f77b8098\" (UID: \"295784c0-89db-405f-b53d-3533f77b8098\") " Nov 03 20:35:15.071346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.071316 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca" pod "295784c0-89db-405f-b53d-3533f77b8098" (UID: "295784c0-89db-405f-b53d-3533f77b8098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:15.071469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.071456 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/295784c0-89db-405f-b53d-3533f77b8098-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:15.072943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.072916 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca" pod "295784c0-89db-405f-b53d-3533f77b8098" (UID: "295784c0-89db-405f-b53d-3533f77b8098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:15.073053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.073037 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4" pod "295784c0-89db-405f-b53d-3533f77b8098" (UID: "295784c0-89db-405f-b53d-3533f77b8098"). InnerVolumeSpecName "kube-api-access-nx7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:15.148327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.148289 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:15.172288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.172255 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx7n4\" (UniqueName: \"kubernetes.io/projected/295784c0-89db-405f-b53d-3533f77b8098-kube-api-access-nx7n4\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:15.172288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.172283 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/295784c0-89db-405f-b53d-3533f77b8098-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:15.286861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.286831 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:15.403903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.403873 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" path="/var/lib/kubelet/pods/9efdcc23-c3fc-447b-92cb-a9decf3ece60/volumes" Nov 03 20:35:15.783356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.783266 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" Nov 03 20:35:15.783356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.783303 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr" event={"ID":"295784c0-89db-405f-b53d-3533f77b8098","Type":"ContainerDied","Data":"2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf"} Nov 03 20:35:15.783356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.783354 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf32f6644526b968adb7e73bdd6c375d64448f01854f8dc7e06c05e82f40ecf" Nov 03 20:35:15.929921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.929892 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:15.977769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.977729 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:15.981984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.981962 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\") pod \"13687593-3523-4ed6-8770-43766068baa6\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " Nov 03 20:35:15.982147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.981993 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\" (UniqueName: \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\") pod \"13687593-3523-4ed6-8770-43766068baa6\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " Nov 03 20:35:15.982147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.982053 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\") pod \"13687593-3523-4ed6-8770-43766068baa6\" (UID: \"13687593-3523-4ed6-8770-43766068baa6\") " Nov 03 20:35:15.982404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.982377 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca" pod "13687593-3523-4ed6-8770-43766068baa6" (UID: "13687593-3523-4ed6-8770-43766068baa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:15.982624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.982601 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13687593-3523-4ed6-8770-43766068baa6-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:15.984282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.984256 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca" pod "13687593-3523-4ed6-8770-43766068baa6" (UID: "13687593-3523-4ed6-8770-43766068baa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:15.984376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:15.984310 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q" pod "13687593-3523-4ed6-8770-43766068baa6" (UID: "13687593-3523-4ed6-8770-43766068baa6"). InnerVolumeSpecName "kube-api-access-8rp4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:16.083987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.083904 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rp4q\" (UniqueName: \"kubernetes.io/projected/13687593-3523-4ed6-8770-43766068baa6-kube-api-access-8rp4q\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:16.083987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.083936 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13687593-3523-4ed6-8770-43766068baa6-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:16.326309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.326262 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" podUID="df041dbc-872e-4479-9b4d-12f3c153dce0" containerName="payload-processing" containerID="cri-o://5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f" gracePeriod=25 Nov 03 20:35:16.583288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.583262 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:35:16.691848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.691810 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\") pod \"df041dbc-872e-4479-9b4d-12f3c153dce0\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " Nov 03 20:35:16.692018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.691980 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\" (UniqueName: \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\") pod \"df041dbc-872e-4479-9b4d-12f3c153dce0\" (UID: \"df041dbc-872e-4479-9b4d-12f3c153dce0\") " Nov 03 20:35:16.692340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.692310 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume" pod "df041dbc-872e-4479-9b4d-12f3c153dce0" (UID: "df041dbc-872e-4479-9b4d-12f3c153dce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:16.694072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.694052 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j" pod "df041dbc-872e-4479-9b4d-12f3c153dce0" (UID: "df041dbc-872e-4479-9b4d-12f3c153dce0"). InnerVolumeSpecName "kube-api-access-6sn6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:16.783130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.783073 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:16.790910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.790880 5954 generic.go:356] "Generic (PLEG): container finished" podID="df041dbc-872e-4479-9b4d-12f3c153dce0" containerID="5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f" exitCode=0 Nov 03 20:35:16.791057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.790929 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" Nov 03 20:35:16.791057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.790959 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" event={"ID":"df041dbc-872e-4479-9b4d-12f3c153dce0","Type":"ContainerDied","Data":"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f"} Nov 03 20:35:16.791057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.791013 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj" event={"ID":"df041dbc-872e-4479-9b4d-12f3c153dce0","Type":"ContainerDied","Data":"a6c23f00c0858dcb4408ecfc1d65e8aef872244a5e9b4f5efeb769bdbe6ab950"} Nov 03 20:35:16.791057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.791041 5954 scope.go:122] "RemoveContainer" containerID="5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f" Nov 03 20:35:16.792576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.792550 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" event={"ID":"13687593-3523-4ed6-8770-43766068baa6","Type":"ContainerDied","Data":"c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063"} Nov 03 20:35:16.792696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.792601 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5800877686d269b317fb07812dc351e81fd34f14057c103e67738a971a1d063" Nov 03 20:35:16.792696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.792632 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt" Nov 03 20:35:16.793172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.793135 5954 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df041dbc-872e-4479-9b4d-12f3c153dce0-plugins-config-volume\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:16.793172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.793165 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn6j\" (UniqueName: \"kubernetes.io/projected/df041dbc-872e-4479-9b4d-12f3c153dce0-kube-api-access-6sn6j\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:16.800772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.800650 5954 scope.go:122] "RemoveContainer" containerID="5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f" Nov 03 20:35:16.801073 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:16.801048 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f\": container with ID starting with 5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f not found: ID does not exist" containerID="5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f" Nov 03 20:35:16.801250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.801083 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f"} err="failed to get container status \"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f\": rpc error: code = NotFound desc = could not find container \"5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f\": container with ID starting with 5e87fd88d65166e0b2ae95daa0e602d1df28f5e1991a175619fe598df24ff13f not found: ID does not exist" Nov 03 20:35:16.845138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.844193 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:35:16.848540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:16.848269 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a6f98b-6c547dcbdd-mhqgj"] Nov 03 20:35:17.404572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:17.404529 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df041dbc-872e-4479-9b4d-12f3c153dce0" path="/var/lib/kubelet/pods/df041dbc-872e-4479-9b4d-12f3c153dce0/volumes" Nov 03 20:35:17.498078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:17.498044 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.190865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.190831 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.446451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.446369 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.678857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.678821 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.997296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.997248 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl"] Nov 03 20:35:18.998195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998161 5954 cpu_manager.go:414] "containerMap: removing container" podUID="13687593-3523-4ed6-8770-43766068baa6" containerName="revoke-keys" Nov 03 20:35:18.998195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998189 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13687593-3523-4ed6-8770-43766068baa6" containerName="revoke-keys" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998218 5954 cpu_manager.go:414] "containerMap: removing container" podUID="df041dbc-872e-4479-9b4d-12f3c153dce0" containerName="payload-processing" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998227 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df041dbc-872e-4479-9b4d-12f3c153dce0" containerName="payload-processing" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998253 5954 cpu_manager.go:414] "containerMap: removing container" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998262 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998298 5954 cpu_manager.go:414] "containerMap: removing container" podUID="295784c0-89db-405f-b53d-3533f77b8098" containerName="revoke-keys" Nov 03 20:35:18.998399 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998307 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295784c0-89db-405f-b53d-3533f77b8098" containerName="revoke-keys" Nov 03 20:35:18.998736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998537 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="13687593-3523-4ed6-8770-43766068baa6" containerName="revoke-keys" Nov 03 20:35:18.998736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998556 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="df041dbc-872e-4479-9b4d-12f3c153dce0" containerName="payload-processing" Nov 03 20:35:18.998736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998572 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efdcc23-c3fc-447b-92cb-a9decf3ece60" containerName="main" Nov 03 20:35:18.998736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:18.998584 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="295784c0-89db-405f-b53d-3533f77b8098" containerName="revoke-keys" Nov 03 20:35:19.005595 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.004547 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.020610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.020580 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:19.020780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.020640 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:35:19.023470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.023442 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl"] Nov 03 20:35:19.123832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.123785 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.124027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.123860 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thrsp\" (UniqueName: \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.124027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.124013 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.166544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.166507 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:19.235053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.234995 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.235274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.235177 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.235274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.235248 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thrsp\" (UniqueName: \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.237172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.236736 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.242357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.241403 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.248448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.248387 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thrsp\" (UniqueName: \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\") pod \"maas-api-revoke-keys-e2e-inf-a-45e569-c7krl\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.340447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.340414 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:19.628072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.628041 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl"] Nov 03 20:35:19.629626 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:19.629595 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba1501c_79bc_4608_9b34_7f8b61c95e3c.slice/crio-5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b WatchSource:0}: Error finding container 5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b: Status 404 returned error can't find the container with id 5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b Nov 03 20:35:19.790636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.790605 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:19.815584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.815547 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" event={"ID":"1ba1501c-79bc-4608-9b34-7f8b61c95e3c","Type":"ContainerStarted","Data":"5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b"} Nov 03 20:35:19.959989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:19.959946 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.105238 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.105163 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg"] Nov 03 20:35:20.112920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.112883 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3aba31-64rqg"] Nov 03 20:35:20.151474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.151441 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.251981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.251884 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.529496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.529404 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.848726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.848626 5954 generic.go:356] "Generic (PLEG): container finished" podID="1ba1501c-79bc-4608-9b34-7f8b61c95e3c" containerID="337e57a2ac33d121a930d1eacad78290f6b531d6c58c768ce4f5d65eefe36c61" exitCode=0 Nov 03 20:35:20.849230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:20.848871 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" event={"ID":"1ba1501c-79bc-4608-9b34-7f8b61c95e3c","Type":"ContainerDied","Data":"337e57a2ac33d121a930d1eacad78290f6b531d6c58c768ce4f5d65eefe36c61"} Nov 03 20:35:21.008755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:21.008722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:21.406430 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:21.406399 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2" path="/var/lib/kubelet/pods/5e6c6f31-5afb-41bf-8c85-6f81e4cc8ea2/volumes" Nov 03 20:35:21.906960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:21.906927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:22.057380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.057355 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:22.189607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.189524 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\") pod \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " Nov 03 20:35:22.189607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.189583 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\" (UniqueName: \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\") pod \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " Nov 03 20:35:22.189817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.189677 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\") pod \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\" (UID: \"1ba1501c-79bc-4608-9b34-7f8b61c95e3c\") " Nov 03 20:35:22.190032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.190005 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca" pod "1ba1501c-79bc-4608-9b34-7f8b61c95e3c" (UID: "1ba1501c-79bc-4608-9b34-7f8b61c95e3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:22.191675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.191649 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca" pod "1ba1501c-79bc-4608-9b34-7f8b61c95e3c" (UID: "1ba1501c-79bc-4608-9b34-7f8b61c95e3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:22.191770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.191740 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp" pod "1ba1501c-79bc-4608-9b34-7f8b61c95e3c" (UID: "1ba1501c-79bc-4608-9b34-7f8b61c95e3c"). InnerVolumeSpecName "kube-api-access-thrsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:22.291586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.291548 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.291586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.291579 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thrsp\" (UniqueName: \"kubernetes.io/projected/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-kube-api-access-thrsp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.291586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.291590 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ba1501c-79bc-4608-9b34-7f8b61c95e3c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:22.799363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.799326 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:22.866909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.866876 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" Nov 03 20:35:22.866909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.866899 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl" event={"ID":"1ba1501c-79bc-4608-9b34-7f8b61c95e3c","Type":"ContainerDied","Data":"5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b"} Nov 03 20:35:22.867177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:22.866956 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7df771613eed488fe16cd8e56db4ed19c54cb9b274f4d561ceea143a9f569b" Nov 03 20:35:23.095987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:23.095905 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:23.723959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:23.723926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:24.573472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:24.573434 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:25.411433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:25.411391 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.030312 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.030281 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.128763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.128730 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.236871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.236837 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.338910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.338010 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:35:26.338910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.338286 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" containerID="cri-o://2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441" gracePeriod=30 Nov 03 20:35:26.389348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:26.389309 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:27.101972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:27.101936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:27.728357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:27.728323 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:27.828205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:27.828174 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:28.066725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:28.066628 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:35:28.067648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:28.067607 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" containerID="cri-o://a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7" gracePeriod=30 Nov 03 20:35:28.104418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:28.104376 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:28.604208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:28.604166 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.139739 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.139703 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.244357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.244317 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.345860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.345823 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.439423 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.439340 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:35:29.439622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.439596 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" podUID="acff8dd6-d623-4be2-b6c3-4262087956ae" containerName="maas-api" containerID="cri-o://b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40" gracePeriod=30 Nov 03 20:35:29.480925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.480896 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.535703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.535666 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z"] Nov 03 20:35:29.541675 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.541648 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a6f98b-z2j5z"] Nov 03 20:35:29.589877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.589840 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.744213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.734640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.744213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.734883 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:35:29.787826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.787757 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\") pod \"acff8dd6-d623-4be2-b6c3-4262087956ae\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " Nov 03 20:35:29.788019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.787875 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\") pod \"acff8dd6-d623-4be2-b6c3-4262087956ae\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " Nov 03 20:35:29.788019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.788004 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\") pod \"acff8dd6-d623-4be2-b6c3-4262087956ae\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " Nov 03 20:35:29.788213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.788086 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\" (UniqueName: \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\") pod \"acff8dd6-d623-4be2-b6c3-4262087956ae\" (UID: \"acff8dd6-d623-4be2-b6c3-4262087956ae\") " Nov 03 20:35:29.788275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.788206 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca" pod "acff8dd6-d623-4be2-b6c3-4262087956ae" (UID: "acff8dd6-d623-4be2-b6c3-4262087956ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:29.788540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.788513 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acff8dd6-d623-4be2-b6c3-4262087956ae-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.790700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.790627 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs" pod "acff8dd6-d623-4be2-b6c3-4262087956ae" (UID: "acff8dd6-d623-4be2-b6c3-4262087956ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:29.790700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.790666 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls" pod "acff8dd6-d623-4be2-b6c3-4262087956ae" (UID: "acff8dd6-d623-4be2-b6c3-4262087956ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:29.791499 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.791467 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc" pod "acff8dd6-d623-4be2-b6c3-4262087956ae" (UID: "acff8dd6-d623-4be2-b6c3-4262087956ae"). InnerVolumeSpecName "kube-api-access-k8rtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:29.848707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.848672 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.892472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.892322 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.898177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.898083 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acff8dd6-d623-4be2-b6c3-4262087956ae-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.898177 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.898146 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rtc\" (UniqueName: \"kubernetes.io/projected/acff8dd6-d623-4be2-b6c3-4262087956ae-kube-api-access-k8rtc\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.907254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.907196 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl"] Nov 03 20:35:29.907254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.907262 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-45e569-c7krl"] Nov 03 20:35:29.916136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.916075 5954 generic.go:356] "Generic (PLEG): container finished" podID="acff8dd6-d623-4be2-b6c3-4262087956ae" containerID="b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40" exitCode=0 Nov 03 20:35:29.916300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.916146 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" Nov 03 20:35:29.916300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.916163 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" event={"ID":"acff8dd6-d623-4be2-b6c3-4262087956ae","Type":"ContainerDied","Data":"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40"} Nov 03 20:35:29.916300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.916224 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn" event={"ID":"acff8dd6-d623-4be2-b6c3-4262087956ae","Type":"ContainerDied","Data":"a0a675f12e0cf04dde63050924b1ee55b88552d245f4de583291841575fdbf4b"} Nov 03 20:35:29.916300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.916253 5954 scope.go:122] "RemoveContainer" containerID="b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40" Nov 03 20:35:29.927361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.927338 5954 scope.go:122] "RemoveContainer" containerID="b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40" Nov 03 20:35:29.927667 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:29.927645 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40\": container with ID starting with b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40 not found: ID does not exist" containerID="b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40" Nov 03 20:35:29.927728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.927677 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40"} err="failed to get container status \"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40\": rpc error: code = NotFound desc = could not find container \"b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40\": container with ID starting with b2b0962081ff8cca4fe917bc6610e0c180008406f32e727ff01c4525b4804d40 not found: ID does not exist" Nov 03 20:35:29.940965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.940931 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:35:29.946193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:29.946165 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-45e569-8787f9468-lkfxn"] Nov 03 20:35:30.008226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:30.008122 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:30.108592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:30.108560 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:30.258954 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:30.258873 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:35:30.806304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:30.806249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:31.404711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.404672 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba1501c-79bc-4608-9b34-7f8b61c95e3c" path="/var/lib/kubelet/pods/1ba1501c-79bc-4608-9b34-7f8b61c95e3c/volumes" Nov 03 20:35:31.405278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.405191 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acff8dd6-d623-4be2-b6c3-4262087956ae" path="/var/lib/kubelet/pods/acff8dd6-d623-4be2-b6c3-4262087956ae/volumes" Nov 03 20:35:31.405604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.405586 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37c7f13-9fd9-4632-8d04-0221d9c0f05c" path="/var/lib/kubelet/pods/d37c7f13-9fd9-4632-8d04-0221d9c0f05c/volumes" Nov 03 20:35:31.493987 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.493956 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:31.617391 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.617327 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:35:31.718429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718397 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718449 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718513 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718568 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718621 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718687 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718776 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718805 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.718881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718841 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\") pod \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\" (UID: \"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957\") " Nov 03 20:35:31.719097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718889 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.719097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.718931 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.719097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719052 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:31.719288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719126 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.719288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719192 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.719376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719359 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.719376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719374 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.719457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719385 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.719457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719400 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.719457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.719419 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.721008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.720985 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:31.721008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.720990 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "kube-api-access-jwt7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.721202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.721194 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.721265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.721206 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy" pod "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" (UID: "c7fadff0-23ad-4fb0-8d19-4da6b7ec6957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.820967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.820929 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.820967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.820960 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.820967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.820971 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.821252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.820980 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwt7n\" (UniqueName: \"kubernetes.io/projected/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957-kube-api-access-jwt7n\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.933073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.932981 5954 generic.go:356] "Generic (PLEG): container finished" podID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerID="2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441" exitCode=0 Nov 03 20:35:31.933073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.933033 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" Nov 03 20:35:31.933300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.933060 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerDied","Data":"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441"} Nov 03 20:35:31.933300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.933159 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c" event={"ID":"c7fadff0-23ad-4fb0-8d19-4da6b7ec6957","Type":"ContainerDied","Data":"9c860dfe60696c05ba1f5e1b1fe943f076b573aba35a6e1788d0ebb9f30f8579"} Nov 03 20:35:31.933300 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.933189 5954 scope.go:122] "RemoveContainer" containerID="2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441" Nov 03 20:35:31.944893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.944874 5954 scope.go:122] "RemoveContainer" containerID="7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10" Nov 03 20:35:31.954255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.954235 5954 scope.go:122] "RemoveContainer" containerID="2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441" Nov 03 20:35:31.954553 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:31.954532 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441\": container with ID starting with 2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441 not found: ID does not exist" containerID="2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441" Nov 03 20:35:31.954625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.954562 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441"} err="failed to get container status \"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441\": rpc error: code = NotFound desc = could not find container \"2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441\": container with ID starting with 2158a309605b6ed4af198211696f2e774e03bac2cc7b1d40246aed58c73a1441 not found: ID does not exist" Nov 03 20:35:31.954625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.954582 5954 scope.go:122] "RemoveContainer" containerID="7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10" Nov 03 20:35:31.954842 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:31.954824 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10\": container with ID starting with 7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10 not found: ID does not exist" containerID="7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10" Nov 03 20:35:31.954886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.954849 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10"} err="failed to get container status \"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10\": rpc error: code = NotFound desc = could not find container \"7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10\": container with ID starting with 7f749d3cff46b7688da61fbb8617372acb2e7d699d0c37f47c7dbb0923daaf10 not found: ID does not exist" Nov 03 20:35:31.959466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.959445 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:35:31.963941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:31.963917 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-zfx4c"] Nov 03 20:35:32.292432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:32.292340 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.986921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:32.986882 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:33.340263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.340234 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:35:33.404455 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.404414 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" path="/var/lib/kubelet/pods/c7fadff0-23ad-4fb0-8d19-4da6b7ec6957/volumes" Nov 03 20:35:33.441165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441130 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441179 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441206 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441230 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441268 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441310 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441344 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441410 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441445 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\") pod \"aeb994dc-361f-460a-8b12-6ad197b08849\" (UID: \"aeb994dc-361f-460a-8b12-6ad197b08849\") " Nov 03 20:35:33.441698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441443 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.441698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441566 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.441698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441642 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.441963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441784 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.441963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.441945 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:33.442086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.442010 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.442086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.442030 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.442086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.442045 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.442086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.442058 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.443590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.443566 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.443590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.443592 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "kube-api-access-4jbv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.443772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.443641 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:33.443772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.443671 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy" pod "aeb994dc-361f-460a-8b12-6ad197b08849" (UID: "aeb994dc-361f-460a-8b12-6ad197b08849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.543057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.542974 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aeb994dc-361f-460a-8b12-6ad197b08849-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.543057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.543006 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbv5\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-kube-api-access-4jbv5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.543057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.543015 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aeb994dc-361f-460a-8b12-6ad197b08849-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.543057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.543025 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aeb994dc-361f-460a-8b12-6ad197b08849-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.543057 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.543033 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aeb994dc-361f-460a-8b12-6ad197b08849-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.702308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.702271 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:33.948208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.948174 5954 generic.go:356] "Generic (PLEG): container finished" podID="aeb994dc-361f-460a-8b12-6ad197b08849" containerID="a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7" exitCode=0 Nov 03 20:35:33.948435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.948232 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" Nov 03 20:35:33.948435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.948288 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" event={"ID":"aeb994dc-361f-460a-8b12-6ad197b08849","Type":"ContainerDied","Data":"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7"} Nov 03 20:35:33.948435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.948339 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65" event={"ID":"aeb994dc-361f-460a-8b12-6ad197b08849","Type":"ContainerDied","Data":"35d18ea80fd27c5a1de369cffb2a1cf7c2540e542def9505479a040c5a6ca39f"} Nov 03 20:35:33.948435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.948364 5954 scope.go:122] "RemoveContainer" containerID="a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7" Nov 03 20:35:33.961101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.961081 5954 scope.go:122] "RemoveContainer" containerID="a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7" Nov 03 20:35:33.961373 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:33.961350 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7\": container with ID starting with a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7 not found: ID does not exist" containerID="a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7" Nov 03 20:35:33.961442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.961382 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7"} err="failed to get container status \"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7\": rpc error: code = NotFound desc = could not find container \"a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7\": container with ID starting with a8775ccb1753046f232370f8a66d1a65fae4db059bd3110f29a34a5678ee5ed7 not found: ID does not exist" Nov 03 20:35:33.983945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.983905 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:35:33.984153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:33.983968 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-d5h65"] Nov 03 20:35:34.393608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:34.393571 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.146553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:35.146493 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.345171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:35.345136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.405569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:35.405492 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" path="/var/lib/kubelet/pods/aeb994dc-361f-460a-8b12-6ad197b08849/volumes" Nov 03 20:35:36.174052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:36.174022 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:35:36.201877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:36.201841 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:37.093340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:37.093289 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.000454 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.000420 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.327871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.327782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.571974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.571936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.686411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.686373 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.929666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.923815 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:35:38.929666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.924065 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" containerID="cri-o://403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c" gracePeriod=30 Nov 03 20:35:38.965030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:38.964941 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:39.599258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:39.599222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:40.407175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:40.407136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:40.509184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:40.509144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:41.100976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:41.100940 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:41.895433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:41.895389 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:42.733041 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:42.733006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.114993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:43.114957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.220896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:43.220862 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.601931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:43.601902 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:44.193248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.193183 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:35:44.284563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284528 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284627 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284701 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284741 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284762 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284830 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284855 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284894 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.284944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284930 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\") pod \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\" (UID: \"f8acdae3-b9d5-4a0d-890c-47e5705643ae\") " Nov 03 20:35:44.285185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285053 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.285185 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.284974 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:44.285356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285331 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.285428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285312 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.285428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285368 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.285428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285404 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.285428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.285418 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.287717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.287689 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:44.287849 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.287829 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.287931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.287911 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.288843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.288812 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv" pod "f8acdae3-b9d5-4a0d-890c-47e5705643ae" (UID: "f8acdae3-b9d5-4a0d-890c-47e5705643ae"). InnerVolumeSpecName "kube-api-access-wfvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.387096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387055 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387096 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387128 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387144 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387159 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvkv\" (UniqueName: \"kubernetes.io/projected/f8acdae3-b9d5-4a0d-890c-47e5705643ae-kube-api-access-wfvkv\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387186 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8acdae3-b9d5-4a0d-890c-47e5705643ae-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.387348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.387200 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8acdae3-b9d5-4a0d-890c-47e5705643ae-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.405674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.405640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:44.513527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:44.513443 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:45.021576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.021536 5954 generic.go:356] "Generic (PLEG): container finished" podID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerID="403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c" exitCode=0 Nov 03 20:35:45.021740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.021584 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" Nov 03 20:35:45.021740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.021621 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" event={"ID":"f8acdae3-b9d5-4a0d-890c-47e5705643ae","Type":"ContainerDied","Data":"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c"} Nov 03 20:35:45.021740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.021676 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q" event={"ID":"f8acdae3-b9d5-4a0d-890c-47e5705643ae","Type":"ContainerDied","Data":"1c2e362545d7ab7895c17f6f1b98376b3b9695893116628caa8925aee8f7edc4"} Nov 03 20:35:45.021740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.021699 5954 scope.go:122] "RemoveContainer" containerID="403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c" Nov 03 20:35:45.038756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.038564 5954 scope.go:122] "RemoveContainer" containerID="403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c" Nov 03 20:35:45.038931 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:45.038907 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c\": container with ID starting with 403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c not found: ID does not exist" containerID="403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c" Nov 03 20:35:45.038995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.038940 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c"} err="failed to get container status \"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c\": rpc error: code = NotFound desc = could not find container \"403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c\": container with ID starting with 403b4c29a5631dbcb2fa62c459a156abdcaa8e2537e36d17f230669cb543f64c not found: ID does not exist" Nov 03 20:35:45.050782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.050753 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:35:45.054877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.054853 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-drn5q"] Nov 03 20:35:45.284692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.284614 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:45.404068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.404036 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" path="/var/lib/kubelet/pods/f8acdae3-b9d5-4a0d-890c-47e5705643ae/volumes" Nov 03 20:35:45.499410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:45.499364 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.001661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:46.001627 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.756267 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:46.756234 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.867350 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:46.867313 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.968469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:46.968440 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:47.141085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.141043 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:35:47.141607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.141571 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" containerID="cri-o://ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e" gracePeriod=30 Nov 03 20:35:47.213097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.213064 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:47.613431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613395 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d"] Nov 03 20:35:47.613883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613867 5954 cpu_manager.go:414] "containerMap: removing container" podUID="1ba1501c-79bc-4608-9b34-7f8b61c95e3c" containerName="revoke-keys" Nov 03 20:35:47.613934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613885 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba1501c-79bc-4608-9b34-7f8b61c95e3c" containerName="revoke-keys" Nov 03 20:35:47.613934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613898 5954 cpu_manager.go:414] "containerMap: removing container" podUID="acff8dd6-d623-4be2-b6c3-4262087956ae" containerName="maas-api" Nov 03 20:35:47.613934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613904 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acff8dd6-d623-4be2-b6c3-4262087956ae" containerName="maas-api" Nov 03 20:35:47.613934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613919 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" Nov 03 20:35:47.613934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613924 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613943 5954 cpu_manager.go:414] "containerMap: removing container" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613947 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613962 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613967 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613978 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.613982 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614044 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614052 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba1501c-79bc-4608-9b34-7f8b61c95e3c" containerName="revoke-keys" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614058 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="acff8dd6-d623-4be2-b6c3-4262087956ae" containerName="maas-api" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614066 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8acdae3-b9d5-4a0d-890c-47e5705643ae" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614076 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb994dc-361f-460a-8b12-6ad197b08849" containerName="istio-proxy" Nov 03 20:35:47.614146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.614083 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fadff0-23ad-4fb0-8d19-4da6b7ec6957" containerName="istio-proxy" Nov 03 20:35:47.619339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.619318 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.626861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.626832 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d"] Nov 03 20:35:47.628641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.627809 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:35:47.628641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.627903 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:35:47.628641 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.628228 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:35:47.633169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.633077 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:35:47.641381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.641354 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.641529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.641394 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.641529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.641422 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mpp\" (UniqueName: \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.742143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.742004 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.742143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.742067 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.742143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.742145 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mpp\" (UniqueName: \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.742828 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.742808 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.744449 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.744426 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.754941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.754916 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mpp\" (UniqueName: \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\") pod \"maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.937788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.937751 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:47.984847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:47.984816 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:48.216558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:48.216523 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d"] Nov 03 20:35:48.219243 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:35:48.219213 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e6b89_452e_4e86_88ec_b6a658dcc10f.slice/crio-9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43 WatchSource:0}: Error finding container 9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43: Status 404 returned error can't find the container with id 9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43 Nov 03 20:35:48.886493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:48.886457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:48.987655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:48.987620 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:49.054651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:49.054616 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" event={"ID":"628e6b89-452e-4e86-88ec-b6a658dcc10f","Type":"ContainerStarted","Data":"8932a9abde14c75f3af6222dda52bc214d0ebc46717a6060f046c782d9c3be5e"} Nov 03 20:35:49.054809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:49.054680 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" event={"ID":"628e6b89-452e-4e86-88ec-b6a658dcc10f","Type":"ContainerStarted","Data":"9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43"} Nov 03 20:35:49.075832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:49.075775 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" podStartSLOduration=2.075759957 podStartE2EDuration="2.075759957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:49.073563627 +0000 UTC m=+2289.798571223" watchObservedRunningTime="2026-11-03 20:35:49.075759957 +0000 UTC m=+2289.800767552" Nov 03 20:35:49.597012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:49.596967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:50.063221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:50.063188 5954 generic.go:356] "Generic (PLEG): container finished" podID="628e6b89-452e-4e86-88ec-b6a658dcc10f" containerID="8932a9abde14c75f3af6222dda52bc214d0ebc46717a6060f046c782d9c3be5e" exitCode=0 Nov 03 20:35:50.063619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:50.063274 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" event={"ID":"628e6b89-452e-4e86-88ec-b6a658dcc10f","Type":"ContainerDied","Data":"8932a9abde14c75f3af6222dda52bc214d0ebc46717a6060f046c782d9c3be5e"} Nov 03 20:35:50.296931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:50.296892 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:50.821869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:50.821836 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:50.997743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:50.997689 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:51.211299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.211271 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:51.278425 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.278391 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\") pod \"628e6b89-452e-4e86-88ec-b6a658dcc10f\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " Nov 03 20:35:51.278622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.278432 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\" (UniqueName: \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\") pod \"628e6b89-452e-4e86-88ec-b6a658dcc10f\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " Nov 03 20:35:51.278622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.278559 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\") pod \"628e6b89-452e-4e86-88ec-b6a658dcc10f\" (UID: \"628e6b89-452e-4e86-88ec-b6a658dcc10f\") " Nov 03 20:35:51.278817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.278794 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca" pod "628e6b89-452e-4e86-88ec-b6a658dcc10f" (UID: "628e6b89-452e-4e86-88ec-b6a658dcc10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:51.278989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.278973 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.280650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.280617 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca" pod "628e6b89-452e-4e86-88ec-b6a658dcc10f" (UID: "628e6b89-452e-4e86-88ec-b6a658dcc10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:51.280805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.280733 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp" pod "628e6b89-452e-4e86-88ec-b6a658dcc10f" (UID: "628e6b89-452e-4e86-88ec-b6a658dcc10f"). InnerVolumeSpecName "kube-api-access-r2mpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:51.380616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.380534 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2mpp\" (UniqueName: \"kubernetes.io/projected/628e6b89-452e-4e86-88ec-b6a658dcc10f-kube-api-access-r2mpp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.380616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.380565 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/628e6b89-452e-4e86-88ec-b6a658dcc10f-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:51.694684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:51.694597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:52.077985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.077901 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" Nov 03 20:35:52.077985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.077916 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d" event={"ID":"628e6b89-452e-4e86-88ec-b6a658dcc10f","Type":"ContainerDied","Data":"9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43"} Nov 03 20:35:52.077985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.077970 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3662e1002eeecb3f87ccf83998414dfd2a05db66de0caaa2593cdddaf16f43" Nov 03 20:35:52.400552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.400524 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:52.412195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.412171 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:35:52.491593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491556 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.491782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491604 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.491782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491639 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.491782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491706 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.491782 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491756 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.492002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491798 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.492002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491874 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.492002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491938 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.492002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.491981 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\") pod \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\" (UID: \"b8840ebc-c04f-48b4-b9af-0048a52ec74e\") " Nov 03 20:35:52.492257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492003 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.492257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492006 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.492257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492065 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.492257 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492076 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.492466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492349 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:52.492526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492474 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.492526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492499 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.492526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492513 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.492709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492527 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.492709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.492540 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.494009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.493975 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:52.494160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.494003 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "kube-api-access-2cmks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:52.494222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.494184 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:52.494318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.494301 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token" pod "b8840ebc-c04f-48b4-b9af-0048a52ec74e" (UID: "b8840ebc-c04f-48b4-b9af-0048a52ec74e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:52.593811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.593772 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.593811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.593804 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.593811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.593815 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:52.594073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:52.593826 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmks\" (UniqueName: \"kubernetes.io/projected/b8840ebc-c04f-48b4-b9af-0048a52ec74e-kube-api-access-2cmks\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:53.085608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.085556 5954 generic.go:356] "Generic (PLEG): container finished" podID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerID="ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e" exitCode=0 Nov 03 20:35:53.085801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.085617 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" Nov 03 20:35:53.085801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.085628 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerDied","Data":"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e"} Nov 03 20:35:53.085801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.085694 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h" event={"ID":"b8840ebc-c04f-48b4-b9af-0048a52ec74e","Type":"ContainerDied","Data":"3a1880f46bfe91c83433e87f19a34bfba53396d5416308323118030c319956cc"} Nov 03 20:35:53.085801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.085725 5954 scope.go:122] "RemoveContainer" containerID="ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e" Nov 03 20:35:53.096450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.096430 5954 scope.go:122] "RemoveContainer" containerID="e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60" Nov 03 20:35:53.104250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.104222 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:53.109931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.109906 5954 scope.go:122] "RemoveContainer" containerID="ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e" Nov 03 20:35:53.110403 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:53.110373 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e\": container with ID starting with ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e not found: ID does not exist" containerID="ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e" Nov 03 20:35:53.110509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.110416 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e"} err="failed to get container status \"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e\": rpc error: code = NotFound desc = could not find container \"ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e\": container with ID starting with ce0389e4b2df250ae2836222ea9bc87e6c36e6d9b65fda2e21b60aeaa11e9e7e not found: ID does not exist" Nov 03 20:35:53.110509 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.110441 5954 scope.go:122] "RemoveContainer" containerID="e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60" Nov 03 20:35:53.110788 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:35:53.110761 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60\": container with ID starting with e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60 not found: ID does not exist" containerID="e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60" Nov 03 20:35:53.110854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.110801 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60"} err="failed to get container status \"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60\": rpc error: code = NotFound desc = could not find container \"e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60\": container with ID starting with e05e150c3ca7bcd4af50bb23136af040ae3b579753e0e75041508f70640b0f60 not found: ID does not exist" Nov 03 20:35:53.112075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.112001 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:35:53.115020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.114995 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-45e569-openshift-default-677d57777f-mzd8h"] Nov 03 20:35:53.403777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.403740 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" path="/var/lib/kubelet/pods/b8840ebc-c04f-48b4-b9af-0048a52ec74e/volumes" Nov 03 20:35:53.898172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:53.898131 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:54.595669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:54.595635 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.289845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:55.289802 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.764313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:55.764274 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.862260 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:55.862213 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.991169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:55.991132 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.735310 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:56.735272 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.388431 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:57.388400 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.798440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:57.798348 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.898542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:57.898512 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.089411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.089327 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.255005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.254967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.440912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.440879 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.456937 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.456907 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d"] Nov 03 20:35:58.459991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.459968 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c60d2a-m7k6d"] Nov 03 20:35:58.611783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.611750 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.979127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:58.979085 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:59.394713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:59.394679 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:59.406254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:35:59.406228 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628e6b89-452e-4e86-88ec-b6a658dcc10f" path="/var/lib/kubelet/pods/628e6b89-452e-4e86-88ec-b6a658dcc10f/volumes" Nov 03 20:36:00.095536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:00.095495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:00.258085 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:00.258048 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:36:00.652857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:00.652822 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:36:00.881244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:00.881208 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:01.596283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:01.596255 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:02.301023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:02.300987 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:03.021724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:03.021685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:03.698261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:03.698221 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:04.405213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:04.405177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.187085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.187052 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.394781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.394752 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.498997 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.498913 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.768703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.768595 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:36:05.768901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.768870 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" containerID="cri-o://93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d" gracePeriod=30 Nov 03 20:36:05.821594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:05.821557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:06.178581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.178546 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:06.297695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.297658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:06.456979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.456899 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl"] Nov 03 20:36:06.457408 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457392 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457410 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457430 5954 cpu_manager.go:414] "containerMap: removing container" podUID="628e6b89-452e-4e86-88ec-b6a658dcc10f" containerName="revoke-keys" Nov 03 20:36:06.457468 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457436 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628e6b89-452e-4e86-88ec-b6a658dcc10f" containerName="revoke-keys" Nov 03 20:36:06.457575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457468 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457474 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457545 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457555 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8840ebc-c04f-48b4-b9af-0048a52ec74e" containerName="istio-proxy" Nov 03 20:36:06.457575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.457563 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="628e6b89-452e-4e86-88ec-b6a658dcc10f" containerName="revoke-keys" Nov 03 20:36:06.460762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.460745 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.468920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.468896 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:36:06.469133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.469047 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:36:06.473101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.473079 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl"] Nov 03 20:36:06.490434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.490405 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:06.490588 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.490452 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:36:06.560864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.560834 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74xq\" (UniqueName: \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.561302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.560971 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.561302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.561027 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.662358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.662321 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.662549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.662377 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.662549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.662469 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74xq\" (UniqueName: \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.662990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.662962 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.664853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.664828 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.675363 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.675336 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74xq\" (UniqueName: \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\") pod \"maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:06.775136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:06.775037 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:07.053529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:07.053503 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl"] Nov 03 20:36:07.055524 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:36:07.055498 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a23dfa9_883b_4c44_9342_832fe7bb4537.slice/crio-c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e WatchSource:0}: Error finding container c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e: Status 404 returned error can't find the container with id c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e Nov 03 20:36:07.165783 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:07.165748 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" event={"ID":"4a23dfa9-883b-4c44-9342-832fe7bb4537","Type":"ContainerStarted","Data":"c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e"} Nov 03 20:36:07.180947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:07.180916 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:07.993528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:07.993496 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:08.173401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:08.173316 5954 generic.go:356] "Generic (PLEG): container finished" podID="4a23dfa9-883b-4c44-9342-832fe7bb4537" containerID="76acfabfc35ea34d2945160405268f158ac3468266434dd691bb6a3a835775ef" exitCode=0 Nov 03 20:36:08.173401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:08.173362 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" event={"ID":"4a23dfa9-883b-4c44-9342-832fe7bb4537","Type":"ContainerDied","Data":"76acfabfc35ea34d2945160405268f158ac3468266434dd691bb6a3a835775ef"} Nov 03 20:36:08.688119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:08.688083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:09.321999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.321974 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:09.406701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.406666 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:09.492103 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.492025 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\" (UniqueName: \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\") pod \"4a23dfa9-883b-4c44-9342-832fe7bb4537\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " Nov 03 20:36:09.492103 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.492069 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\") pod \"4a23dfa9-883b-4c44-9342-832fe7bb4537\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " Nov 03 20:36:09.492331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.492184 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\") pod \"4a23dfa9-883b-4c44-9342-832fe7bb4537\" (UID: \"4a23dfa9-883b-4c44-9342-832fe7bb4537\") " Nov 03 20:36:09.492475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.492448 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca" pod "4a23dfa9-883b-4c44-9342-832fe7bb4537" (UID: "4a23dfa9-883b-4c44-9342-832fe7bb4537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:09.492589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.492564 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:09.494295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.494276 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq" pod "4a23dfa9-883b-4c44-9342-832fe7bb4537" (UID: "4a23dfa9-883b-4c44-9342-832fe7bb4537"). InnerVolumeSpecName "kube-api-access-d74xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:09.494379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.494326 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca" pod "4a23dfa9-883b-4c44-9342-832fe7bb4537" (UID: "4a23dfa9-883b-4c44-9342-832fe7bb4537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:09.593802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.593763 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74xq\" (UniqueName: \"kubernetes.io/projected/4a23dfa9-883b-4c44-9342-832fe7bb4537-kube-api-access-d74xq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:09.593802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:09.593796 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a23dfa9-883b-4c44-9342-832fe7bb4537-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:10.100294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:10.100236 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:10.185827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:10.185785 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" event={"ID":"4a23dfa9-883b-4c44-9342-832fe7bb4537","Type":"ContainerDied","Data":"c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e"} Nov 03 20:36:10.185979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:10.185836 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e0ac99e84752d288ec30b5fd7cbe5d717629fd721729d72d32e7b335a7696e" Nov 03 20:36:10.185979 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:10.185794 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl" Nov 03 20:36:10.800752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:10.800726 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:11.022823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.022797 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:36:11.108124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108081 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108145 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108185 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108224 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108243 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108271 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108427 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108499 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.108602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108526 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.108602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108548 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.108789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108698 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.108789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108735 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.108789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108773 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\") pod \"6989da16-dbc6-425e-94f5-39b8f2777431\" (UID: \"6989da16-dbc6-425e-94f5-39b8f2777431\") " Nov 03 20:36:11.109021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.108997 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:11.109159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.109142 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.109241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.109160 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6989da16-dbc6-425e-94f5-39b8f2777431-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.109241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.109169 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.109241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.109177 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.109241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.109185 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.110577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.110546 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:11.110687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.110673 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:11.110879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.110861 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:11.110942 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.110878 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn" pod "6989da16-dbc6-425e-94f5-39b8f2777431" (UID: "6989da16-dbc6-425e-94f5-39b8f2777431"). InnerVolumeSpecName "kube-api-access-l8wsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:11.193581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.193542 5954 generic.go:356] "Generic (PLEG): container finished" podID="6989da16-dbc6-425e-94f5-39b8f2777431" containerID="93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d" exitCode=0 Nov 03 20:36:11.193750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.193593 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" Nov 03 20:36:11.193750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.193628 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerDied","Data":"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d"} Nov 03 20:36:11.193750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.193686 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f" event={"ID":"6989da16-dbc6-425e-94f5-39b8f2777431","Type":"ContainerDied","Data":"95c4a0fcbe87bae124204b3e676835e3d89deaf945cb44f4ede124740964b3a9"} Nov 03 20:36:11.193750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.193717 5954 scope.go:122] "RemoveContainer" containerID="93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d" Nov 03 20:36:11.203798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.203782 5954 scope.go:122] "RemoveContainer" containerID="803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7" Nov 03 20:36:11.209902 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.209883 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6989da16-dbc6-425e-94f5-39b8f2777431-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.209998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.209903 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.209998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.209914 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6989da16-dbc6-425e-94f5-39b8f2777431-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.209998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.209923 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wsn\" (UniqueName: \"kubernetes.io/projected/6989da16-dbc6-425e-94f5-39b8f2777431-kube-api-access-l8wsn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:11.212478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.212465 5954 scope.go:122] "RemoveContainer" containerID="93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d" Nov 03 20:36:11.212731 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:11.212705 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d\": container with ID starting with 93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d not found: ID does not exist" containerID="93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d" Nov 03 20:36:11.212796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.212740 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d"} err="failed to get container status \"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d\": rpc error: code = NotFound desc = could not find container \"93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d\": container with ID starting with 93cf58f5a17a683b2454bd445d2e1fb5c4130fa82127a78ab2d71b26f61c734d not found: ID does not exist" Nov 03 20:36:11.212796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.212757 5954 scope.go:122] "RemoveContainer" containerID="803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7" Nov 03 20:36:11.213011 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:11.212995 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7\": container with ID starting with 803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7 not found: ID does not exist" containerID="803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7" Nov 03 20:36:11.213063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.213017 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7"} err="failed to get container status \"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7\": rpc error: code = NotFound desc = could not find container \"803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7\": container with ID starting with 803cb253d48036608e9a4117351db97c718aa73fb1ccda6484913a10dbdf7ab7 not found: ID does not exist" Nov 03 20:36:11.228024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.228001 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:36:11.233137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.233101 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c60d2a-openshift-default-6db7dfff49-kdp7f"] Nov 03 20:36:11.258548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.258527 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:36:11.267083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.267057 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:36:11.403256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.403144 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:11.404263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.404238 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" path="/var/lib/kubelet/pods/6989da16-dbc6-425e-94f5-39b8f2777431/volumes" Nov 03 20:36:11.544045 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:11.544004 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.192243 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:12.192209 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.888670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:12.888624 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.676642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:13.676606 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:14.384978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:14.384936 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.094353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:15.094312 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.793895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:15.793855 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.504814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:16.504777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.732407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:16.732368 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.944978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:16.944919 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:17.120274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120240 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4"] Nov 03 20:36:17.120777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120756 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.120886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120779 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.120886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120799 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.120886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120806 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.120886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120824 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4a23dfa9-883b-4c44-9342-832fe7bb4537" containerName="revoke-keys" Nov 03 20:36:17.120886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120833 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a23dfa9-883b-4c44-9342-832fe7bb4537" containerName="revoke-keys" Nov 03 20:36:17.121173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120960 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.121173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120974 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6989da16-dbc6-425e-94f5-39b8f2777431" containerName="istio-proxy" Nov 03 20:36:17.121173 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.120985 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a23dfa9-883b-4c44-9342-832fe7bb4537" containerName="revoke-keys" Nov 03 20:36:17.126131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.126071 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.133559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.133527 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4"] Nov 03 20:36:17.136602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.136581 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:17.136602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.136584 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:36:17.136776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.136606 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:36:17.136776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.136606 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:36:17.137203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.137181 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g"] Nov 03 20:36:17.141428 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.141410 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.149609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.149588 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g"] Nov 03 20:36:17.178288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178258 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.178458 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178305 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.178514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178461 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgwq\" (UniqueName: \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.178514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178495 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.178629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178600 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4f6g\" (UniqueName: \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.178693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.178662 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.279916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.279830 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.279916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.279880 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.280180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.279917 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.280180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.279987 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgwq\" (UniqueName: \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.280180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.280014 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.280180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.280056 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4f6g\" (UniqueName: \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.280668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.280640 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.280766 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.280706 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.282432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.282407 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.282517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.282430 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.293672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.293647 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgwq\" (UniqueName: \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\") pod \"maas-api-revoke-sub-b814e8f1-r87m4\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.293775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.293648 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4f6g\" (UniqueName: \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\") pod \"maas-api-revoke-sub-0d828537-99c8g\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.443279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.443244 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:17.458429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.458406 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:17.728906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.728881 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4"] Nov 03 20:36:17.731174 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:36:17.731138 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98907a5f_88f4_488c_8932_dcec6dabc6bf.slice/crio-0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7 WatchSource:0}: Error finding container 0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7: Status 404 returned error can't find the container with id 0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7 Nov 03 20:36:17.747251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:17.747224 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g"] Nov 03 20:36:17.749121 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:36:17.749085 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa850e2_42d4_44f6_8db7_cffb2d70d54b.slice/crio-4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7 WatchSource:0}: Error finding container 4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7: Status 404 returned error can't find the container with id 4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7 Nov 03 20:36:18.237639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.237542 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" event={"ID":"3aa850e2-42d4-44f6-8db7-cffb2d70d54b","Type":"ContainerStarted","Data":"49d7991892906598b554347636367fcbceb35ef2847cff588a1f3a43d4a21e78"} Nov 03 20:36:18.237639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.237601 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" event={"ID":"3aa850e2-42d4-44f6-8db7-cffb2d70d54b","Type":"ContainerStarted","Data":"4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7"} Nov 03 20:36:18.239247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.239209 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" event={"ID":"98907a5f-88f4-488c-8932-dcec6dabc6bf","Type":"ContainerStarted","Data":"bb073bd8d73ea0585f02eb58862e208f43ebab003fbd4d4c43d5be542d50b8ef"} Nov 03 20:36:18.239403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.239255 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" event={"ID":"98907a5f-88f4-488c-8932-dcec6dabc6bf","Type":"ContainerStarted","Data":"0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7"} Nov 03 20:36:18.256176 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.256102 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" podStartSLOduration=1.256086658 podStartE2EDuration="1.256086658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:18.255861633 +0000 UTC m=+2318.980869242" watchObservedRunningTime="2026-11-03 20:36:18.256086658 +0000 UTC m=+2318.981094255" Nov 03 20:36:18.277624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:18.277580 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" podStartSLOduration=1.277568668 podStartE2EDuration="1.277568668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:18.275878299 +0000 UTC m=+2319.000885897" watchObservedRunningTime="2026-11-03 20:36:18.277568668 +0000 UTC m=+2319.002576266" Nov 03 20:36:19.246179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:19.246142 5954 generic.go:356] "Generic (PLEG): container finished" podID="3aa850e2-42d4-44f6-8db7-cffb2d70d54b" containerID="49d7991892906598b554347636367fcbceb35ef2847cff588a1f3a43d4a21e78" exitCode=0 Nov 03 20:36:19.246630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:19.246227 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" event={"ID":"3aa850e2-42d4-44f6-8db7-cffb2d70d54b","Type":"ContainerDied","Data":"49d7991892906598b554347636367fcbceb35ef2847cff588a1f3a43d4a21e78"} Nov 03 20:36:19.247775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:19.247749 5954 generic.go:356] "Generic (PLEG): container finished" podID="98907a5f-88f4-488c-8932-dcec6dabc6bf" containerID="bb073bd8d73ea0585f02eb58862e208f43ebab003fbd4d4c43d5be542d50b8ef" exitCode=0 Nov 03 20:36:19.247905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:19.247800 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" event={"ID":"98907a5f-88f4-488c-8932-dcec6dabc6bf","Type":"ContainerDied","Data":"bb073bd8d73ea0585f02eb58862e208f43ebab003fbd4d4c43d5be542d50b8ef"} Nov 03 20:36:20.392268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.392240 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:20.418419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.417472 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\") pod \"98907a5f-88f4-488c-8932-dcec6dabc6bf\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " Nov 03 20:36:20.418419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.417606 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\" (UniqueName: \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\") pod \"98907a5f-88f4-488c-8932-dcec6dabc6bf\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " Nov 03 20:36:20.418419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.417711 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\") pod \"98907a5f-88f4-488c-8932-dcec6dabc6bf\" (UID: \"98907a5f-88f4-488c-8932-dcec6dabc6bf\") " Nov 03 20:36:20.418742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.418533 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca" pod "98907a5f-88f4-488c-8932-dcec6dabc6bf" (UID: "98907a5f-88f4-488c-8932-dcec6dabc6bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:20.420609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.420573 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq" pod "98907a5f-88f4-488c-8932-dcec6dabc6bf" (UID: "98907a5f-88f4-488c-8932-dcec6dabc6bf"). InnerVolumeSpecName "kube-api-access-hhgwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:20.423079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.422683 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca" pod "98907a5f-88f4-488c-8932-dcec6dabc6bf" (UID: "98907a5f-88f4-488c-8932-dcec6dabc6bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:20.443077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.443055 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:20.518946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.518873 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\") pod \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " Nov 03 20:36:20.518946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.518915 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\" (UniqueName: \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\") pod \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " Nov 03 20:36:20.519184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.518994 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\") pod \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\" (UID: \"3aa850e2-42d4-44f6-8db7-cffb2d70d54b\") " Nov 03 20:36:20.519401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.519370 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:20.519401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.519392 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98907a5f-88f4-488c-8932-dcec6dabc6bf-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:20.519576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.519408 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhgwq\" (UniqueName: \"kubernetes.io/projected/98907a5f-88f4-488c-8932-dcec6dabc6bf-kube-api-access-hhgwq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:20.519576 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.519431 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca" pod "3aa850e2-42d4-44f6-8db7-cffb2d70d54b" (UID: "3aa850e2-42d4-44f6-8db7-cffb2d70d54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:20.520966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.520944 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca" pod "3aa850e2-42d4-44f6-8db7-cffb2d70d54b" (UID: "3aa850e2-42d4-44f6-8db7-cffb2d70d54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:20.521024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.521001 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g" pod "3aa850e2-42d4-44f6-8db7-cffb2d70d54b" (UID: "3aa850e2-42d4-44f6-8db7-cffb2d70d54b"). InnerVolumeSpecName "kube-api-access-p4f6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:20.620789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.620758 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:20.620789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.620785 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4f6g\" (UniqueName: \"kubernetes.io/projected/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-kube-api-access-p4f6g\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:20.620789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:20.620796 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aa850e2-42d4-44f6-8db7-cffb2d70d54b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:21.261027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.260996 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" Nov 03 20:36:21.261251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.261028 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g" event={"ID":"3aa850e2-42d4-44f6-8db7-cffb2d70d54b","Type":"ContainerDied","Data":"4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7"} Nov 03 20:36:21.261251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.261120 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8e1fe68d1b4570b4eeb42f0f857a72c69a72c4eda973a3c6c6253b148009e7" Nov 03 20:36:21.262621 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.262602 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" Nov 03 20:36:21.262735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.262645 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4" event={"ID":"98907a5f-88f4-488c-8932-dcec6dabc6bf","Type":"ContainerDied","Data":"0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7"} Nov 03 20:36:21.262735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:21.262686 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c285dca7baeaf6cf7657b334935b45be429c4519809bd95142b5f7aba4274a7" Nov 03 20:36:22.982356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:22.982321 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:26.279304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.279261 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" containerID="cri-o://3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf" gracePeriod=45 Nov 03 20:36:26.291517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.291484 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" containerID="cri-o://b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8" gracePeriod=45 Nov 03 20:36:26.368984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.368955 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Nov 03 20:36:26.545212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.545149 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:36:26.567852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.567827 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:36:26.579130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579077 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579199 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\" (UniqueName: \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579239 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579266 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579370 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.579457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579385 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579442 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\") pod \"15ac93b8-880b-4a16-bc42-ab30525a26af\" (UID: \"15ac93b8-880b-4a16-bc42-ab30525a26af\") " Nov 03 20:36:26.579870 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579843 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.580036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.579935 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.580130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.580080 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.581787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.581762 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "kube-api-access-7k2bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:26.581886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.581770 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:26.582418 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.582397 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm" pod "15ac93b8-880b-4a16-bc42-ab30525a26af" (UID: "15ac93b8-880b-4a16-bc42-ab30525a26af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.681476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681439 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681503 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681536 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681609 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681651 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\" (UniqueName: \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681691 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\") pod \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\" (UID: \"c4654d15-6a0d-4eb5-aae2-696f5209aa0c\") " Nov 03 20:36:26.681891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681709 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.681891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681789 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.681891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.681851 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.682071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682053 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682066 5954 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-home\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682077 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ac93b8-880b-4a16-bc42-ab30525a26af-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682088 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k2bl\" (UniqueName: \"kubernetes.io/projected/15ac93b8-880b-4a16-bc42-ab30525a26af-kube-api-access-7k2bl\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682097 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682121 5954 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tmp-dir\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682129 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.682202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.682137 5954 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ac93b8-880b-4a16-bc42-ab30525a26af-model-cache\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.683638 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.683615 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.683672 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "kube-api-access-gpfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:26.683752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.683675 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs" pod "c4654d15-6a0d-4eb5-aae2-696f5209aa0c" (UID: "c4654d15-6a0d-4eb5-aae2-696f5209aa0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:26.783092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.783059 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpfgv\" (UniqueName: \"kubernetes.io/projected/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-kube-api-access-gpfgv\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.783092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.783087 5954 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-dshm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.783092 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:26.783097 5954 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4654d15-6a0d-4eb5-aae2-696f5209aa0c-tls-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:27.299404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.299370 5954 generic.go:356] "Generic (PLEG): container finished" podID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerID="3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf" exitCode=0 Nov 03 20:36:27.299790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.299425 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" Nov 03 20:36:27.299790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.299458 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" event={"ID":"15ac93b8-880b-4a16-bc42-ab30525a26af","Type":"ContainerDied","Data":"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf"} Nov 03 20:36:27.299790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.299507 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv" event={"ID":"15ac93b8-880b-4a16-bc42-ab30525a26af","Type":"ContainerDied","Data":"e8f2697cbed78887f9dc6b48c544b0297cc3a320c370e01d57934e946d4b874e"} Nov 03 20:36:27.299790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.299523 5954 scope.go:122] "RemoveContainer" containerID="3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf" Nov 03 20:36:27.300949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.300924 5954 generic.go:356] "Generic (PLEG): container finished" podID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerID="b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8" exitCode=0 Nov 03 20:36:27.301040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.300974 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" Nov 03 20:36:27.301147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.301097 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" event={"ID":"c4654d15-6a0d-4eb5-aae2-696f5209aa0c","Type":"ContainerDied","Data":"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8"} Nov 03 20:36:27.301272 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.301159 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n" event={"ID":"c4654d15-6a0d-4eb5-aae2-696f5209aa0c","Type":"ContainerDied","Data":"cbb2dc7489c9f767eced3ee6f6e7cc17d48c3ea6d27ccc5204ae5e05999df06a"} Nov 03 20:36:27.310833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.310717 5954 scope.go:122] "RemoveContainer" containerID="3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf" Nov 03 20:36:27.311076 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:27.311055 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf\": container with ID starting with 3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf not found: ID does not exist" containerID="3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf" Nov 03 20:36:27.311188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.311082 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf"} err="failed to get container status \"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf\": rpc error: code = NotFound desc = could not find container \"3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf\": container with ID starting with 3960c3d6a169727e9e09b4df674b6636f82cad02e28cbce7a73579e96b167ddf not found: ID does not exist" Nov 03 20:36:27.311188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.311098 5954 scope.go:122] "RemoveContainer" containerID="b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8" Nov 03 20:36:27.319651 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.319627 5954 scope.go:122] "RemoveContainer" containerID="b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8" Nov 03 20:36:27.319909 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:27.319890 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8\": container with ID starting with b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8 not found: ID does not exist" containerID="b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8" Nov 03 20:36:27.319976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.319921 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8"} err="failed to get container status \"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8\": rpc error: code = NotFound desc = could not find container \"b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8\": container with ID starting with b4b33476e8941fc2a496079fa7e678758fbd5751dbdef2b5e32e96bbdd4af8e8 not found: ID does not exist" Nov 03 20:36:27.325482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.325461 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:36:27.335877 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.335858 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-w0-b1b190-kserve-c75c78b9b-zzjrv"] Nov 03 20:36:27.348659 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.348637 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:36:27.356735 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.356711 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/premium-w0-b1b190-kserve-8c796b64f-jgg6n"] Nov 03 20:36:27.403427 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.403400 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" path="/var/lib/kubelet/pods/15ac93b8-880b-4a16-bc42-ab30525a26af/volumes" Nov 03 20:36:27.403879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:27.403865 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" path="/var/lib/kubelet/pods/c4654d15-6a0d-4eb5-aae2-696f5209aa0c/volumes" Nov 03 20:36:29.523889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:29.523856 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:36:30.257406 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:36:30.257370 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:36:32.220091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.220060 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.368079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.368046 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.375939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.375912 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl"] Nov 03 20:36:32.380876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.380849 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b1b190-nlnnl"] Nov 03 20:36:32.470157 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.470055 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.928982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:32.928950 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:33.403155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:33.403124 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a23dfa9-883b-4c44-9342-832fe7bb4537" path="/var/lib/kubelet/pods/4a23dfa9-883b-4c44-9342-832fe7bb4537/volumes" Nov 03 20:36:35.041251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:35.041209 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:36:38.809497 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:38.809466 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:38.910380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:38.910349 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:39.145290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:39.145256 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:36:39.145798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:39.145772 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" containerID="cri-o://cc1de6d805a8fbdcdcf253c45d5cb94f7f40499ed6539c64269a7f00dae68cbc" gracePeriod=30 Nov 03 20:36:39.202922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:39.202885 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:42.656136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.656091 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:36:42.890887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.890854 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:36:42.891480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891464 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" Nov 03 20:36:42.891533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891483 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" Nov 03 20:36:42.891533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891512 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3aa850e2-42d4-44f6-8db7-cffb2d70d54b" containerName="revoke-keys" Nov 03 20:36:42.891533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891520 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa850e2-42d4-44f6-8db7-cffb2d70d54b" containerName="revoke-keys" Nov 03 20:36:42.891533 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891529 5954 cpu_manager.go:414] "containerMap: removing container" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" Nov 03 20:36:42.891673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891537 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" Nov 03 20:36:42.891673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891576 5954 cpu_manager.go:414] "containerMap: removing container" podUID="98907a5f-88f4-488c-8932-dcec6dabc6bf" containerName="revoke-keys" Nov 03 20:36:42.891673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891583 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98907a5f-88f4-488c-8932-dcec6dabc6bf" containerName="revoke-keys" Nov 03 20:36:42.891673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891660 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4654d15-6a0d-4eb5-aae2-696f5209aa0c" containerName="main" Nov 03 20:36:42.891673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891673 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa850e2-42d4-44f6-8db7-cffb2d70d54b" containerName="revoke-keys" Nov 03 20:36:42.891838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891682 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ac93b8-880b-4a16-bc42-ab30525a26af" containerName="main" Nov 03 20:36:42.891838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.891695 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="98907a5f-88f4-488c-8932-dcec6dabc6bf" containerName="revoke-keys" Nov 03 20:36:42.895381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.895361 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:42.903487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.903451 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64wg7\"" Nov 03 20:36:42.904071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.904042 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:36:42.904246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.904161 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:36:42.904246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.904232 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:36:42.906319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.906271 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:36:42.961911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.961882 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcvx\" (UniqueName: \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:42.962091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.961918 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:42.962091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:42.962034 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.062772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.062735 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcvx\" (UniqueName: \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.062945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.062793 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.062945 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.062884 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.065078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.065054 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.065209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.065126 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.074585 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.074563 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcvx\" (UniqueName: \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\") pod \"maas-controller-6dd9f77dd9-8w77m\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.211846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.211752 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:43.494704 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.494679 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:36:43.496776 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:36:43.496751 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85021ba3_424b_41b2_aeae_45bfcc32d2d4.slice/crio-deada7155760de2a841a8a82c1d2ca7181a8c37bb1d4addeb8bfafeafe2ab067 WatchSource:0}: Error finding container deada7155760de2a841a8a82c1d2ca7181a8c37bb1d4addeb8bfafeafe2ab067: Status 404 returned error can't find the container with id deada7155760de2a841a8a82c1d2ca7181a8c37bb1d4addeb8bfafeafe2ab067 Nov 03 20:36:43.498515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:43.498489 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:36:44.220438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.220399 5954 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:36:44.220918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.220468 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 20:36:44.403971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.403931 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" event={"ID":"85021ba3-424b-41b2-aeae-45bfcc32d2d4","Type":"ContainerStarted","Data":"deada7155760de2a841a8a82c1d2ca7181a8c37bb1d4addeb8bfafeafe2ab067"} Nov 03 20:36:44.405498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.405434 5954 generic.go:356] "Generic (PLEG): container finished" podID="dc59581e-6842-470a-a017-335297a600e7" containerID="cc1de6d805a8fbdcdcf253c45d5cb94f7f40499ed6539c64269a7f00dae68cbc" exitCode=0 Nov 03 20:36:44.405600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.405506 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" event={"ID":"dc59581e-6842-470a-a017-335297a600e7","Type":"ContainerDied","Data":"cc1de6d805a8fbdcdcf253c45d5cb94f7f40499ed6539c64269a7f00dae68cbc"} Nov 03 20:36:44.423481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.423425 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:36:44.474326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474287 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474373 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474405 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474439 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474473 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474541 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474585 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474646 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474715 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\") pod \"dc59581e-6842-470a-a017-335297a600e7\" (UID: \"dc59581e-6842-470a-a017-335297a600e7\") " Nov 03 20:36:44.474984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.474937 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.475928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.475066 5954 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-data\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.475928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.475653 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.475928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.475738 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.475928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.475772 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.475928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.475892 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:44.477566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.477538 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:44.478887 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.478836 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.479000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.478933 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.480643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.480611 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t" pod "dc59581e-6842-470a-a017-335297a600e7" (UID: "dc59581e-6842-470a-a017-335297a600e7"). InnerVolumeSpecName "kube-api-access-5d89t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576280 5954 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576316 5954 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-workload-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576332 5954 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-istio-envoy\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576346 5954 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc59581e-6842-470a-a017-335297a600e7-istio-podinfo\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576360 5954 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-istio-token\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576375 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d89t\" (UniqueName: \"kubernetes.io/projected/dc59581e-6842-470a-a017-335297a600e7-kube-api-access-5d89t\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576389 5954 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc59581e-6842-470a-a017-335297a600e7-credential-socket\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.576438 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:44.576403 5954 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc59581e-6842-470a-a017-335297a600e7-istiod-ca-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:45.414691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:45.414660 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" Nov 03 20:36:45.415085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:45.414694 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q" event={"ID":"dc59581e-6842-470a-a017-335297a600e7","Type":"ContainerDied","Data":"21ffea51c614cc6afb08ab9fe79820d6c9ec8ae248126f7361eb20fdaebe6c05"} Nov 03 20:36:45.415085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:45.414760 5954 scope.go:122] "RemoveContainer" containerID="cc1de6d805a8fbdcdcf253c45d5cb94f7f40499ed6539c64269a7f00dae68cbc" Nov 03 20:36:45.436386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:45.436362 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:36:45.440713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:45.440691 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-g2v7q"] Nov 03 20:36:46.424557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:46.424521 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" event={"ID":"85021ba3-424b-41b2-aeae-45bfcc32d2d4","Type":"ContainerStarted","Data":"3ac6a060b45009485711f6f5225a98483c21c00c03b0fab048ec0eab618e2c8e"} Nov 03 20:36:46.425005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:46.424662 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:36:46.446523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:46.446473 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" podStartSLOduration=2.450623745 podStartE2EDuration="4.446460297s" totalImagesPullingTime="1.995836552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:46.445569338 +0000 UTC m=+2347.170576935" watchObservedRunningTime="2026-11-03 20:36:46.446460297 +0000 UTC m=+2347.171467870" Nov 03 20:36:47.403456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.403423 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc59581e-6842-470a-a017-335297a600e7" path="/var/lib/kubelet/pods/dc59581e-6842-470a-a017-335297a600e7/volumes" Nov 03 20:36:47.446246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.446215 5954 scope.go:122] "RemoveContainer" containerID="3c2f5826274cb208e909d4b0aa8d62d8ed42ffedc5dcbc8dc7475b46b7e09fc8" Nov 03 20:36:47.456422 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.456399 5954 scope.go:122] "RemoveContainer" containerID="2f9f96cc5f00f3d387b5951bded21650f58d1276b0a8198405b2c0a07f46421c" Nov 03 20:36:47.464679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.464656 5954 scope.go:122] "RemoveContainer" containerID="ccc0266ff835b80f94fd1959bb624251ea742ec66b3c0fe3bcc20e4a8cf7d1c0" Nov 03 20:36:47.829756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.829658 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:47.936406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:47.936372 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:48.047710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:48.047648 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:48.146213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:48.146182 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:48.996779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:48.996740 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:49.098609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:49.098574 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:54.678761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:54.678679 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.432906 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:36:57.432871 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:37:00.256934 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:37:00.256888 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:37:03.602798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.602761 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:03.705062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.705021 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:03.808349 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.808311 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc"] Nov 03 20:37:03.808827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.808797 5954 cpu_manager.go:414] "containerMap: removing container" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" Nov 03 20:37:03.808827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.808822 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" Nov 03 20:37:03.809026 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.808953 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc59581e-6842-470a-a017-335297a600e7" containerName="istio-proxy" Nov 03 20:37:03.814086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.814066 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.818570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.818462 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:37:03.818970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.818948 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:03.819305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.819284 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:37:03.819420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.819284 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:37:03.822464 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.822442 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc"] Nov 03 20:37:03.868127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.868027 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.868127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.868068 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gg2\" (UniqueName: \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.868353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.868207 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.968805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.968773 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.969002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.968823 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gg2\" (UniqueName: \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.969002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.968854 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.969549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.969521 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.971655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.971634 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:03.981975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:03.981949 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gg2\" (UniqueName: \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\") pod \"maas-api-revoke-sub-85237e45-7jzfc\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:04.131232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:04.131140 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:04.410961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:04.410896 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc"] Nov 03 20:37:04.413430 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:37:04.413400 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a04fd3_48c9_421b_9574_2acd9461bd76.slice/crio-e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37 WatchSource:0}: Error finding container e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37: Status 404 returned error can't find the container with id e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37 Nov 03 20:37:04.531728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:04.531690 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" event={"ID":"64a04fd3-48c9-421b-9574-2acd9461bd76","Type":"ContainerStarted","Data":"e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37"} Nov 03 20:37:05.539087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:05.539002 5954 generic.go:356] "Generic (PLEG): container finished" podID="64a04fd3-48c9-421b-9574-2acd9461bd76" containerID="995bbbe4f1d11e3147c31020d1a6742a225eb7b47a22dabde1b4d74de4a4b7ac" exitCode=0 Nov 03 20:37:05.539517 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:05.539082 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" event={"ID":"64a04fd3-48c9-421b-9574-2acd9461bd76","Type":"ContainerDied","Data":"995bbbe4f1d11e3147c31020d1a6742a225eb7b47a22dabde1b4d74de4a4b7ac"} Nov 03 20:37:05.650335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:05.650304 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:37:06.682486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.682453 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:06.798722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.798643 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\") pod \"64a04fd3-48c9-421b-9574-2acd9461bd76\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " Nov 03 20:37:06.798912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.798730 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\") pod \"64a04fd3-48c9-421b-9574-2acd9461bd76\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " Nov 03 20:37:06.798912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.798784 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\" (UniqueName: \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\") pod \"64a04fd3-48c9-421b-9574-2acd9461bd76\" (UID: \"64a04fd3-48c9-421b-9574-2acd9461bd76\") " Nov 03 20:37:06.799098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.799072 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca" pod "64a04fd3-48c9-421b-9574-2acd9461bd76" (UID: "64a04fd3-48c9-421b-9574-2acd9461bd76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:06.799212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.799192 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:06.800868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.800849 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca" pod "64a04fd3-48c9-421b-9574-2acd9461bd76" (UID: "64a04fd3-48c9-421b-9574-2acd9461bd76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:06.800982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.800963 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2" pod "64a04fd3-48c9-421b-9574-2acd9461bd76" (UID: "64a04fd3-48c9-421b-9574-2acd9461bd76"). InnerVolumeSpecName "kube-api-access-s9gg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:06.900526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.900492 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64a04fd3-48c9-421b-9574-2acd9461bd76-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:06.900526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:06.900519 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9gg2\" (UniqueName: \"kubernetes.io/projected/64a04fd3-48c9-421b-9574-2acd9461bd76-kube-api-access-s9gg2\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:07.550896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:07.550864 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" Nov 03 20:37:07.550896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:07.550891 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc" event={"ID":"64a04fd3-48c9-421b-9574-2acd9461bd76","Type":"ContainerDied","Data":"e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37"} Nov 03 20:37:07.551126 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:07.550938 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2157a569e542eca2ffffae0b606fd1bd679fa0002db641876ec64bc7cc1cc37" Nov 03 20:37:13.078900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.078864 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt"] Nov 03 20:37:13.079520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.079502 5954 cpu_manager.go:414] "containerMap: removing container" podUID="64a04fd3-48c9-421b-9574-2acd9461bd76" containerName="revoke-keys" Nov 03 20:37:13.079589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.079521 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a04fd3-48c9-421b-9574-2acd9461bd76" containerName="revoke-keys" Nov 03 20:37:13.079633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.079612 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a04fd3-48c9-421b-9574-2acd9461bd76" containerName="revoke-keys" Nov 03 20:37:13.083324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.083301 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.098475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.098445 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl"] Nov 03 20:37:13.103501 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.103473 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt"] Nov 03 20:37:13.103668 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.103585 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.137929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.137890 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl"] Nov 03 20:37:13.167725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.167680 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.167725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.167729 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.167817 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.167917 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.167962 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.168007 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.168027 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddad7abe-544d-4e16-a193-f7c905fe1e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.168049 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxbq\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-kube-api-access-jxxbq\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.168274 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.168175 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269239 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snhb\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-kube-api-access-8snhb\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.269279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269285 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269308 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269383 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddad7abe-544d-4e16-a193-f7c905fe1e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269429 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxbq\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-kube-api-access-jxxbq\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269484 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269537 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269561 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269590 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269757 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.269900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269815 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.269900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269858 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.270118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269924 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.270118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269932 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270118 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.269994 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270098 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270174 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270193 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddad7abe-544d-4e16-a193-f7c905fe1e64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270214 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270248 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.270283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270259 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270301 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.270549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.270341 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.272346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.272321 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.272456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.272353 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.282076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.282048 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.282376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.282357 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxbq\" (UniqueName: \"kubernetes.io/projected/ddad7abe-544d-4e16-a193-f7c905fe1e64-kube-api-access-jxxbq\") pod \"maas-default-gateway-openshift-default-77f556d647-mf6wt\" (UID: \"ddad7abe-544d-4e16-a193-f7c905fe1e64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.371313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371277 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371321 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371355 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371380 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371412 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371440 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371477 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371510 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snhb\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-kube-api-access-8snhb\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.371855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371655 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.372006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.371975 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.372082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.372009 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.372168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.372079 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.372287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.372266 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.372443 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.372423 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.373793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.373771 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.374278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.374260 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.384313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.384284 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.384511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.384491 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snhb\" (UniqueName: \"kubernetes.io/projected/0fb68c68-b69e-4e9b-bc3f-589b465a1ffa-kube-api-access-8snhb\") pod \"maas-default-gateway-openshift-default-77f556d647-zdxwl\" (UID: \"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.398964 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.398941 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:13.438005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.437967 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:13.684419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.684384 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt"] Nov 03 20:37:13.687980 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:37:13.687946 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddad7abe_544d_4e16_a193_f7c905fe1e64.slice/crio-d787a94ebd2a814c834c1c088d537281c24c5f5f5c6ad0d55aa193f490261f31 WatchSource:0}: Error finding container d787a94ebd2a814c834c1c088d537281c24c5f5f5c6ad0d55aa193f490261f31: Status 404 returned error can't find the container with id d787a94ebd2a814c834c1c088d537281c24c5f5f5c6ad0d55aa193f490261f31 Nov 03 20:37:13.690968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.690937 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.691053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.691008 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.691097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.691056 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.732072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.732043 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl"] Nov 03 20:37:13.734512 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:37:13.734481 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb68c68_b69e_4e9b_bc3f_589b465a1ffa.slice/crio-41b0a64ba8d6c80d91cf2b722748c73bb3e43246e9304bfc507b84dadee3b456 WatchSource:0}: Error finding container 41b0a64ba8d6c80d91cf2b722748c73bb3e43246e9304bfc507b84dadee3b456: Status 404 returned error can't find the container with id 41b0a64ba8d6c80d91cf2b722748c73bb3e43246e9304bfc507b84dadee3b456 Nov 03 20:37:13.736882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.736847 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.736977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.736937 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.737020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:13.736989 5954 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:14.591581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.591536 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" event={"ID":"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa","Type":"ContainerStarted","Data":"9dcbb434fbdd7fa682fd63bcd89a4d955a0a58bc5488bb9525167ae9a1459636"} Nov 03 20:37:14.592020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.591599 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" event={"ID":"0fb68c68-b69e-4e9b-bc3f-589b465a1ffa","Type":"ContainerStarted","Data":"41b0a64ba8d6c80d91cf2b722748c73bb3e43246e9304bfc507b84dadee3b456"} Nov 03 20:37:14.593175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.593151 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" event={"ID":"ddad7abe-544d-4e16-a193-f7c905fe1e64","Type":"ContainerStarted","Data":"1a93f5b5af7e99425642fe7a5af16c02bbc88872751c685b7dc0e915d335422c"} Nov 03 20:37:14.593229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.593193 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" event={"ID":"ddad7abe-544d-4e16-a193-f7c905fe1e64","Type":"ContainerStarted","Data":"d787a94ebd2a814c834c1c088d537281c24c5f5f5c6ad0d55aa193f490261f31"} Nov 03 20:37:14.670128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.668567 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podStartSLOduration=1.668552773 podStartE2EDuration="1.668552773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:14.667672806 +0000 UTC m=+2375.392680401" watchObservedRunningTime="2026-11-03 20:37:14.668552773 +0000 UTC m=+2375.393560365" Nov 03 20:37:14.670128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:14.668893 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podStartSLOduration=1.668886041 podStartE2EDuration="1.668886041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:14.637052793 +0000 UTC m=+2375.362060390" watchObservedRunningTime="2026-11-03 20:37:14.668886041 +0000 UTC m=+2375.393893637" Nov 03 20:37:15.234143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.232818 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:15.355129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.354830 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:37:15.367381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.355081 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" podUID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" containerName="manager" containerID="cri-o://3ac6a060b45009485711f6f5225a98483c21c00c03b0fab048ec0eab618e2c8e" gracePeriod=30 Nov 03 20:37:15.401326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.401292 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:15.401473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.401371 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:15.405064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.405046 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:15.439130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.438470 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:15.453537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.440586 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:15.453537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.440644 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:15.602426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.602390 5954 generic.go:356] "Generic (PLEG): container finished" podID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" containerID="3ac6a060b45009485711f6f5225a98483c21c00c03b0fab048ec0eab618e2c8e" exitCode=0 Nov 03 20:37:15.602810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.602463 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" event={"ID":"85021ba3-424b-41b2-aeae-45bfcc32d2d4","Type":"ContainerDied","Data":"3ac6a060b45009485711f6f5225a98483c21c00c03b0fab048ec0eab618e2c8e"} Nov 03 20:37:15.771188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.771116 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:37:15.897570 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.897528 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\") pod \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " Nov 03 20:37:15.897772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.897638 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\" (UniqueName: \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\") pod \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " Nov 03 20:37:15.897772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.897692 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\") pod \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\" (UID: \"85021ba3-424b-41b2-aeae-45bfcc32d2d4\") " Nov 03 20:37:15.901990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.901953 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx" pod "85021ba3-424b-41b2-aeae-45bfcc32d2d4" (UID: "85021ba3-424b-41b2-aeae-45bfcc32d2d4"). InnerVolumeSpecName "kube-api-access-6gcvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:15.902252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.902231 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert" pod "85021ba3-424b-41b2-aeae-45bfcc32d2d4" (UID: "85021ba3-424b-41b2-aeae-45bfcc32d2d4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:15.902394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.902335 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs" pod "85021ba3-424b-41b2-aeae-45bfcc32d2d4" (UID: "85021ba3-424b-41b2-aeae-45bfcc32d2d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:15.999218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.999175 5954 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-webhook-cert\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:15.999218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.999212 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcvx\" (UniqueName: \"kubernetes.io/projected/85021ba3-424b-41b2-aeae-45bfcc32d2d4-kube-api-access-6gcvx\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:15.999440 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:15.999226 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85021ba3-424b-41b2-aeae-45bfcc32d2d4-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:16.399895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.399856 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:16.400093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.399908 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:16.439132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.438521 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:16.439132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.438584 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:16.610693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.610662 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" Nov 03 20:37:16.610693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.610677 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-8w77m" event={"ID":"85021ba3-424b-41b2-aeae-45bfcc32d2d4","Type":"ContainerDied","Data":"deada7155760de2a841a8a82c1d2ca7181a8c37bb1d4addeb8bfafeafe2ab067"} Nov 03 20:37:16.611158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.610754 5954 scope.go:122] "RemoveContainer" containerID="3ac6a060b45009485711f6f5225a98483c21c00c03b0fab048ec0eab618e2c8e" Nov 03 20:37:16.685012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.684983 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:37:16.689469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:16.689445 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-8w77m"] Nov 03 20:37:17.399971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.399938 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:17.400170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.399990 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:17.405191 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.405167 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" path="/var/lib/kubelet/pods/85021ba3-424b-41b2-aeae-45bfcc32d2d4/volumes" Nov 03 20:37:17.441134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.440594 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:17.441134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.440649 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:17.811514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.811411 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.966134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.960616 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-sxcbm"] Nov 03 20:37:17.966134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.965869 5954 cpu_manager.go:414] "containerMap: removing container" podUID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" containerName="manager" Nov 03 20:37:17.966134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.965897 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" containerName="manager" Nov 03 20:37:17.966134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.966034 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="85021ba3-424b-41b2-aeae-45bfcc32d2d4" containerName="manager" Nov 03 20:37:17.972088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.972064 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:17.976699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.976657 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:37:17.976699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.976677 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:37:17.976925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.976759 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:37:17.977338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.977100 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64wg7\"" Nov 03 20:37:17.978577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:17.978555 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-sxcbm"] Nov 03 20:37:18.073146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.073045 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-webhook-cert\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.073366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.073341 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vzf\" (UniqueName: \"kubernetes.io/projected/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-kube-api-access-p6vzf\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.073515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.073502 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-metrics-certs\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.174755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.174717 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vzf\" (UniqueName: \"kubernetes.io/projected/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-kube-api-access-p6vzf\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.174755 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.174758 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-metrics-certs\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.174977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.174789 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-webhook-cert\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.177339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.177293 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-webhook-cert\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.177634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.177615 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-metrics-certs\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.187292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.187269 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vzf\" (UniqueName: \"kubernetes.io/projected/16c22b88-db5d-4e5f-8ada-5bfb6c7dd972-kube-api-access-p6vzf\") pod \"maas-controller-6dd9f77dd9-sxcbm\" (UID: \"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972\") " pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.289419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.289370 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:18.400164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.400130 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:18.400314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.400190 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:18.442132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.441538 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:18.442132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.441606 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:18.610491 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.610466 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-sxcbm"] Nov 03 20:37:18.611595 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:37:18.611574 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c22b88_db5d_4e5f_8ada_5bfb6c7dd972.slice/crio-64ef364a0295d9f51197ef192730cecbe3b3b4d00ea9df66fc5c02eabfd7c850 WatchSource:0}: Error finding container 64ef364a0295d9f51197ef192730cecbe3b3b4d00ea9df66fc5c02eabfd7c850: Status 404 returned error can't find the container with id 64ef364a0295d9f51197ef192730cecbe3b3b4d00ea9df66fc5c02eabfd7c850 Nov 03 20:37:18.666263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:18.665176 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" event={"ID":"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972","Type":"ContainerStarted","Data":"64ef364a0295d9f51197ef192730cecbe3b3b4d00ea9df66fc5c02eabfd7c850"} Nov 03 20:37:19.400490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.400456 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:19.400947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.400510 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:19.456189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.455518 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:19.456189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.455583 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:19.676130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.674165 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" event={"ID":"16c22b88-db5d-4e5f-8ada-5bfb6c7dd972","Type":"ContainerStarted","Data":"996498714df81998333182047b5c6e6d20d15fca15b34113ded2a58a91231d2d"} Nov 03 20:37:19.676130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.674250 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:19.696807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:19.696755 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" podStartSLOduration=2.111422587 podStartE2EDuration="2.696740674s" totalImagesPullingTime="585.318087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:19.694169357 +0000 UTC m=+2380.419176954" watchObservedRunningTime="2026-11-03 20:37:19.696740674 +0000 UTC m=+2380.421748267" Nov 03 20:37:20.399628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:20.399596 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mf6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:37:20.399803 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:20.399650 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" podUID="ddad7abe-544d-4e16-a193-f7c905fe1e64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 20:37:20.438864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:20.438830 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdxwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 20:37:20.439249 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:20.438884 5954 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" podUID="0fb68c68-b69e-4e9b-bc3f-589b465a1ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 20:37:22.082635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.082595 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:22.083411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.083376 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:22.084396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.084320 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdxwl" Nov 03 20:37:22.084529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.084447 5954 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:22.084838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.084808 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:22.085639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:22.085620 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mf6wt" Nov 03 20:37:30.257401 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:37:30.257360 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:37:30.685178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:30.685146 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd9f77dd9-sxcbm" Nov 03 20:37:49.061814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:49.061780 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:52.969797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:52.969763 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:37:54.150264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:37:54.150227 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:00.257894 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:38:00.257852 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:38:00.291241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.291211 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq"] Nov 03 20:38:00.296578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.296558 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.302078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.302056 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:00.302411 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.302068 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:00.302549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.302126 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:00.302549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.302173 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:38:00.305254 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.305231 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq"] Nov 03 20:38:00.336420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.336393 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvzp\" (UniqueName: \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.336556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.336430 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.336556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.336501 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.437284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.437234 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvzp\" (UniqueName: \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.437284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.437290 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.437508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.437335 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.437948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.437927 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.439669 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.439651 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.449947 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.449926 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvzp\" (UniqueName: \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\") pod \"maas-api-revoke-sub-c1e271a9-mgbfq\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.613448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.613424 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:00.890866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.890839 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq"] Nov 03 20:38:00.893899 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:38:00.893869 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd275d_b440_40d0_a463_8bbe41ba0b13.slice/crio-3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58 WatchSource:0}: Error finding container 3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58: Status 404 returned error can't find the container with id 3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58 Nov 03 20:38:00.930728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:00.930702 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" event={"ID":"84bd275d-b440-40d0-a463-8bbe41ba0b13","Type":"ContainerStarted","Data":"3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58"} Nov 03 20:38:01.937015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:01.936984 5954 generic.go:356] "Generic (PLEG): container finished" podID="84bd275d-b440-40d0-a463-8bbe41ba0b13" containerID="bf092a3a372c71b7e696b0d5de802b475b0feed92ae302cd538610355e91695e" exitCode=0 Nov 03 20:38:01.937414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:01.937119 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" event={"ID":"84bd275d-b440-40d0-a463-8bbe41ba0b13","Type":"ContainerDied","Data":"bf092a3a372c71b7e696b0d5de802b475b0feed92ae302cd538610355e91695e"} Nov 03 20:38:03.076765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.076732 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:03.162532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.162500 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\") pod \"84bd275d-b440-40d0-a463-8bbe41ba0b13\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " Nov 03 20:38:03.162715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.162548 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\") pod \"84bd275d-b440-40d0-a463-8bbe41ba0b13\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " Nov 03 20:38:03.162715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.162573 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\" (UniqueName: \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\") pod \"84bd275d-b440-40d0-a463-8bbe41ba0b13\" (UID: \"84bd275d-b440-40d0-a463-8bbe41ba0b13\") " Nov 03 20:38:03.162921 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.162894 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca" pod "84bd275d-b440-40d0-a463-8bbe41ba0b13" (UID: "84bd275d-b440-40d0-a463-8bbe41ba0b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:03.163074 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.163060 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:03.164586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.164570 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca" pod "84bd275d-b440-40d0-a463-8bbe41ba0b13" (UID: "84bd275d-b440-40d0-a463-8bbe41ba0b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:03.164691 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.164624 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp" pod "84bd275d-b440-40d0-a463-8bbe41ba0b13" (UID: "84bd275d-b440-40d0-a463-8bbe41ba0b13"). InnerVolumeSpecName "kube-api-access-rsvzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:03.264386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.264306 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84bd275d-b440-40d0-a463-8bbe41ba0b13-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:03.264386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.264337 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvzp\" (UniqueName: \"kubernetes.io/projected/84bd275d-b440-40d0-a463-8bbe41ba0b13-kube-api-access-rsvzp\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:03.950629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.950597 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" event={"ID":"84bd275d-b440-40d0-a463-8bbe41ba0b13","Type":"ContainerDied","Data":"3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58"} Nov 03 20:38:03.950786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.950646 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2e2751656d3600ecbb43308c9b9a55cb16d34149944b36b3174bbb721dfa58" Nov 03 20:38:03.950786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:03.950608 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq" Nov 03 20:38:11.255379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:11.255343 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:38:15.265130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:15.258938 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:38:19.049603 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.049567 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld"] Nov 03 20:38:19.050019 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.050005 5954 cpu_manager.go:414] "containerMap: removing container" podUID="84bd275d-b440-40d0-a463-8bbe41ba0b13" containerName="revoke-keys" Nov 03 20:38:19.050070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.050020 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd275d-b440-40d0-a463-8bbe41ba0b13" containerName="revoke-keys" Nov 03 20:38:19.050132 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.050115 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bd275d-b440-40d0-a463-8bbe41ba0b13" containerName="revoke-keys" Nov 03 20:38:19.053833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.053810 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.060851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.060826 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:19.061097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.060835 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:19.061097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.060869 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:19.061293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.060904 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:38:19.063795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.063773 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld"] Nov 03 20:38:19.138832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.138794 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfsj\" (UniqueName: \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.138998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.138845 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.138998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.138953 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.240376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.240336 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfsj\" (UniqueName: \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.240376 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.240382 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.240660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.240434 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.241227 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.241206 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.242829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.242806 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.252364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.252343 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfsj\" (UniqueName: \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\") pod \"maas-api-revoke-sub-2e1f5b5f-lxnld\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.369928 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.369899 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:19.651751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:19.651724 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld"] Nov 03 20:38:19.653649 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:38:19.653607 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d3f304_ce22_4f3d_bf52_984ed7f61dd4.slice/crio-6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb WatchSource:0}: Error finding container 6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb: Status 404 returned error can't find the container with id 6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb Nov 03 20:38:20.076619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:20.076536 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" event={"ID":"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4","Type":"ContainerStarted","Data":"b101841e1d2a3c1ab3362ea1198ad389eaebc572e1ff1806c34dcd979fd199b2"} Nov 03 20:38:20.076619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:20.076593 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" event={"ID":"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4","Type":"ContainerStarted","Data":"6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb"} Nov 03 20:38:20.095679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:20.095632 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" podStartSLOduration=1.095620002 podStartE2EDuration="1.095620002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:20.094596624 +0000 UTC m=+2440.819604220" watchObservedRunningTime="2026-11-03 20:38:20.095620002 +0000 UTC m=+2440.820627597" Nov 03 20:38:21.084852 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:21.084821 5954 generic.go:356] "Generic (PLEG): container finished" podID="b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" containerID="b101841e1d2a3c1ab3362ea1198ad389eaebc572e1ff1806c34dcd979fd199b2" exitCode=0 Nov 03 20:38:21.085340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:21.084904 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" event={"ID":"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4","Type":"ContainerDied","Data":"b101841e1d2a3c1ab3362ea1198ad389eaebc572e1ff1806c34dcd979fd199b2"} Nov 03 20:38:22.252612 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.252582 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:22.370472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.370438 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\" (UniqueName: \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\") pod \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " Nov 03 20:38:22.370672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.370561 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\") pod \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " Nov 03 20:38:22.370672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.370608 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\") pod \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\" (UID: \"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4\") " Nov 03 20:38:22.370931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.370905 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca" pod "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" (UID: "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:22.372610 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.372576 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj" pod "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" (UID: "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4"). InnerVolumeSpecName "kube-api-access-6bfsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:22.372708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.372608 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca" pod "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" (UID: "b0d3f304-ce22-4f3d-bf52-984ed7f61dd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:22.471628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.471593 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bfsj\" (UniqueName: \"kubernetes.io/projected/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-kube-api-access-6bfsj\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:22.471628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.471625 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:22.471891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:22.471639 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:23.097334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:23.097296 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" event={"ID":"b0d3f304-ce22-4f3d-bf52-984ed7f61dd4","Type":"ContainerDied","Data":"6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb"} Nov 03 20:38:23.097334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:23.097327 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld" Nov 03 20:38:23.097545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:23.097353 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f47ad10252d69d9d33d5fc72a60df4ed0aaccb58504ca906d0667ccb1c1eeeb" Nov 03 20:38:30.112216 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.112184 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m"] Nov 03 20:38:30.112679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.112617 5954 cpu_manager.go:414] "containerMap: removing container" podUID="b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" containerName="revoke-keys" Nov 03 20:38:30.112679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.112629 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" containerName="revoke-keys" Nov 03 20:38:30.112765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.112717 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" containerName="revoke-keys" Nov 03 20:38:30.116148 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.116129 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.122512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.122494 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:38:30.123971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.123932 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:38:30.125124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.124652 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:38:30.125124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.125095 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:38:30.125508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.125490 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:38:30.128743 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.128003 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m"] Nov 03 20:38:30.251916 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.251884 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.252084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.252024 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rjs\" (UniqueName: \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.252181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.252165 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.258366 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:38:30.258321 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:38:30.354294 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.354252 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.354493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.354339 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.354493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.354447 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rjs\" (UniqueName: \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.355709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.355677 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.361161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.358617 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.367565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.367455 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rjs\" (UniqueName: \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\") pod \"maas-api-revoke-sub-ec843018-h2f5m\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.449618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.449582 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:30.732636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:30.732612 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m"] Nov 03 20:38:30.734963 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:38:30.734934 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd807bc_59a4_4bee_8096_4445150c7999.slice/crio-fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2 WatchSource:0}: Error finding container fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2: Status 404 returned error can't find the container with id fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2 Nov 03 20:38:31.144767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:31.144724 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" event={"ID":"6dd807bc-59a4-4bee-8096-4445150c7999","Type":"ContainerStarted","Data":"ff1cc1b74019758f87a3149201452d6c3fc63b3a3e6c6f856f8f2274e15dc71e"} Nov 03 20:38:31.145285 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:31.144779 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" event={"ID":"6dd807bc-59a4-4bee-8096-4445150c7999","Type":"ContainerStarted","Data":"fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2"} Nov 03 20:38:31.164171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:31.164102 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" podStartSLOduration=1.164085644 podStartE2EDuration="1.164085644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:31.161461551 +0000 UTC m=+2451.886469146" watchObservedRunningTime="2026-11-03 20:38:31.164085644 +0000 UTC m=+2451.889093241" Nov 03 20:38:32.151568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:32.151536 5954 generic.go:356] "Generic (PLEG): container finished" podID="6dd807bc-59a4-4bee-8096-4445150c7999" containerID="ff1cc1b74019758f87a3149201452d6c3fc63b3a3e6c6f856f8f2274e15dc71e" exitCode=0 Nov 03 20:38:32.151992 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:32.151620 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" event={"ID":"6dd807bc-59a4-4bee-8096-4445150c7999","Type":"ContainerDied","Data":"ff1cc1b74019758f87a3149201452d6c3fc63b3a3e6c6f856f8f2274e15dc71e"} Nov 03 20:38:33.380396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.380349 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:33.492565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.492465 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\") pod \"6dd807bc-59a4-4bee-8096-4445150c7999\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " Nov 03 20:38:33.492764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.492580 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\" (UniqueName: \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\") pod \"6dd807bc-59a4-4bee-8096-4445150c7999\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " Nov 03 20:38:33.492764 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.492673 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\") pod \"6dd807bc-59a4-4bee-8096-4445150c7999\" (UID: \"6dd807bc-59a4-4bee-8096-4445150c7999\") " Nov 03 20:38:33.493512 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.493435 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca" pod "6dd807bc-59a4-4bee-8096-4445150c7999" (UID: "6dd807bc-59a4-4bee-8096-4445150c7999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:33.495931 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.495901 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs" pod "6dd807bc-59a4-4bee-8096-4445150c7999" (UID: "6dd807bc-59a4-4bee-8096-4445150c7999"). InnerVolumeSpecName "kube-api-access-s7rjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:33.496221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.496190 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca" pod "6dd807bc-59a4-4bee-8096-4445150c7999" (UID: "6dd807bc-59a4-4bee-8096-4445150c7999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:33.593643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.593597 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:33.593643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.593631 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rjs\" (UniqueName: \"kubernetes.io/projected/6dd807bc-59a4-4bee-8096-4445150c7999-kube-api-access-s7rjs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:33.593643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:33.593641 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd807bc-59a4-4bee-8096-4445150c7999-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:34.189737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:34.189697 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" event={"ID":"6dd807bc-59a4-4bee-8096-4445150c7999","Type":"ContainerDied","Data":"fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2"} Nov 03 20:38:34.190024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:34.190003 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa248527473de097ebb3bb152999ee0bbfed63a36ed9967310fb8e286a0debf2" Nov 03 20:38:34.190214 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:34.190197 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m" Nov 03 20:38:34.792253 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:34.792213 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:38:34.792761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:34.792285 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:38:42.930337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:42.930304 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:38:49.112198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:49.112164 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:56.022701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:56.022657 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt"] Nov 03 20:38:56.024832 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:56.024795 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c04d0a2f-phcwt"] Nov 03 20:38:57.403723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:38:57.403693 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6759f67-5884-4bf9-a11c-d7abbf7aea70" path="/var/lib/kubelet/pods/f6759f67-5884-4bf9-a11c-d7abbf7aea70/volumes" Nov 03 20:39:00.258533 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:39:00.258492 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:39:04.296930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:04.296884 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:06.599435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.599395 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs"] Nov 03 20:39:06.599919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.599901 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6dd807bc-59a4-4bee-8096-4445150c7999" containerName="revoke-keys" Nov 03 20:39:06.599972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.599920 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd807bc-59a4-4bee-8096-4445150c7999" containerName="revoke-keys" Nov 03 20:39:06.600021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.599997 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd807bc-59a4-4bee-8096-4445150c7999" containerName="revoke-keys" Nov 03 20:39:06.603665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.603645 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.609664 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.609614 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:06.610181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.610096 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:39:06.613218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.613196 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs"] Nov 03 20:39:06.655278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.655245 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.655478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.655290 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.655559 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.655491 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzswg\" (UniqueName: \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.668843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.668817 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:39:06.668985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.668817 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:39:06.756291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.756250 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzswg\" (UniqueName: \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.756484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.756367 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.756484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.756408 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.757102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.757076 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.758768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.758748 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.768550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.768527 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzswg\" (UniqueName: \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\") pod \"maas-api-revoke-sub-046e1343-djdfs\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:06.919951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:06.919919 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:07.195351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:07.195325 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs"] Nov 03 20:39:07.197836 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:39:07.197809 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7818f2c0_904d_41aa_8423_cda90a11d150.slice/crio-9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1 WatchSource:0}: Error finding container 9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1: Status 404 returned error can't find the container with id 9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1 Nov 03 20:39:07.410655 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:07.410618 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" event={"ID":"7818f2c0-904d-41aa-8423-cda90a11d150","Type":"ContainerStarted","Data":"9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1"} Nov 03 20:39:08.418493 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:08.418454 5954 generic.go:356] "Generic (PLEG): container finished" podID="7818f2c0-904d-41aa-8423-cda90a11d150" containerID="9db30041540bc0c011ab598c1121a2d544e558dc3bf50f6e2db68e48c2809257" exitCode=0 Nov 03 20:39:08.418910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:08.418541 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" event={"ID":"7818f2c0-904d-41aa-8423-cda90a11d150","Type":"ContainerDied","Data":"9db30041540bc0c011ab598c1121a2d544e558dc3bf50f6e2db68e48c2809257"} Nov 03 20:39:09.567633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.567606 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:09.686193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.686088 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\") pod \"7818f2c0-904d-41aa-8423-cda90a11d150\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " Nov 03 20:39:09.686368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.686259 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\" (UniqueName: \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\") pod \"7818f2c0-904d-41aa-8423-cda90a11d150\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " Nov 03 20:39:09.686368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.686296 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\") pod \"7818f2c0-904d-41aa-8423-cda90a11d150\" (UID: \"7818f2c0-904d-41aa-8423-cda90a11d150\") " Nov 03 20:39:09.686707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.686681 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca" pod "7818f2c0-904d-41aa-8423-cda90a11d150" (UID: "7818f2c0-904d-41aa-8423-cda90a11d150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:09.688240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.688215 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca" pod "7818f2c0-904d-41aa-8423-cda90a11d150" (UID: "7818f2c0-904d-41aa-8423-cda90a11d150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:09.688360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.688336 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg" pod "7818f2c0-904d-41aa-8423-cda90a11d150" (UID: "7818f2c0-904d-41aa-8423-cda90a11d150"). InnerVolumeSpecName "kube-api-access-jzswg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:09.787245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.787212 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzswg\" (UniqueName: \"kubernetes.io/projected/7818f2c0-904d-41aa-8423-cda90a11d150-kube-api-access-jzswg\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:09.787245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.787238 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:09.787245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:09.787248 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7818f2c0-904d-41aa-8423-cda90a11d150-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:10.431180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:10.431139 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" event={"ID":"7818f2c0-904d-41aa-8423-cda90a11d150","Type":"ContainerDied","Data":"9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1"} Nov 03 20:39:10.431370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:10.431192 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3fcde24dd8cd4a9283aa456707eb3617c09832fc387cafa2fa97858af1d5a1" Nov 03 20:39:10.431370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:10.431157 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs" Nov 03 20:39:16.835939 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.835908 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5"] Nov 03 20:39:16.836413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.836397 5954 cpu_manager.go:414] "containerMap: removing container" podUID="7818f2c0-904d-41aa-8423-cda90a11d150" containerName="revoke-keys" Nov 03 20:39:16.836470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.836414 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7818f2c0-904d-41aa-8423-cda90a11d150" containerName="revoke-keys" Nov 03 20:39:16.836528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.836517 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="7818f2c0-904d-41aa-8423-cda90a11d150" containerName="revoke-keys" Nov 03 20:39:16.840009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.839992 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:16.848062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.848038 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:39:16.848240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.848175 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:39:16.848240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.848200 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:16.848240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.848237 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:39:16.851571 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.851543 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5"] Nov 03 20:39:16.961789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.961752 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:16.961966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.961823 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:16.961966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:16.961907 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8pg\" (UniqueName: \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.062938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.062896 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.063190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.063046 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.063190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.063081 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8pg\" (UniqueName: \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.063724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.063700 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.065583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.065562 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.076338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.076312 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8pg\" (UniqueName: \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\") pod \"maas-api-revoke-sub-d3fba056-tbtk5\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.157676 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.157634 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:17.445883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.445849 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5"] Nov 03 20:39:17.449161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.449134 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:39:17.471990 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:17.471957 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" event={"ID":"3671b5fd-8f95-46af-a40d-35b318d9f03b","Type":"ContainerStarted","Data":"26cd7de7ca2da0716aa4b3b1226fe153923f03f4d077e41d4bc36bea6cac09e1"} Nov 03 20:39:18.482694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:18.482591 5954 generic.go:356] "Generic (PLEG): container finished" podID="3671b5fd-8f95-46af-a40d-35b318d9f03b" containerID="89d6d5ad560f0c3116685db8ba69f3167197b00b76a33072e3e256562dcc465a" exitCode=0 Nov 03 20:39:18.483210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:18.482715 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" event={"ID":"3671b5fd-8f95-46af-a40d-35b318d9f03b","Type":"ContainerDied","Data":"89d6d5ad560f0c3116685db8ba69f3167197b00b76a33072e3e256562dcc465a"} Nov 03 20:39:19.803297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.803200 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:39:19.803297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.803260 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:39:19.883195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.882497 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:19.904968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.904937 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\") pod \"3671b5fd-8f95-46af-a40d-35b318d9f03b\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " Nov 03 20:39:19.905193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.905144 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\" (UniqueName: \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\") pod \"3671b5fd-8f95-46af-a40d-35b318d9f03b\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " Nov 03 20:39:19.905193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.905176 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\") pod \"3671b5fd-8f95-46af-a40d-35b318d9f03b\" (UID: \"3671b5fd-8f95-46af-a40d-35b318d9f03b\") " Nov 03 20:39:19.905383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.905357 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca" pod "3671b5fd-8f95-46af-a40d-35b318d9f03b" (UID: "3671b5fd-8f95-46af-a40d-35b318d9f03b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:19.905513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.905498 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:19.907467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.907439 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg" pod "3671b5fd-8f95-46af-a40d-35b318d9f03b" (UID: "3671b5fd-8f95-46af-a40d-35b318d9f03b"). InnerVolumeSpecName "kube-api-access-5l8pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:19.907587 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:19.907541 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca" pod "3671b5fd-8f95-46af-a40d-35b318d9f03b" (UID: "3671b5fd-8f95-46af-a40d-35b318d9f03b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:20.006847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:20.006809 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8pg\" (UniqueName: \"kubernetes.io/projected/3671b5fd-8f95-46af-a40d-35b318d9f03b-kube-api-access-5l8pg\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:20.006847 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:20.006847 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3671b5fd-8f95-46af-a40d-35b318d9f03b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:20.498305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:20.498263 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" event={"ID":"3671b5fd-8f95-46af-a40d-35b318d9f03b","Type":"ContainerDied","Data":"26cd7de7ca2da0716aa4b3b1226fe153923f03f4d077e41d4bc36bea6cac09e1"} Nov 03 20:39:20.498457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:20.498314 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cd7de7ca2da0716aa4b3b1226fe153923f03f4d077e41d4bc36bea6cac09e1" Nov 03 20:39:20.498457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:20.498280 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5" Nov 03 20:39:27.747012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:27.746980 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:39:30.260969 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:39:30.260932 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:39:32.287989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:32.287948 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:39:37.631382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.631352 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r"] Nov 03 20:39:37.632463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.632435 5954 cpu_manager.go:414] "containerMap: removing container" podUID="3671b5fd-8f95-46af-a40d-35b318d9f03b" containerName="revoke-keys" Nov 03 20:39:37.632463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.632459 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3671b5fd-8f95-46af-a40d-35b318d9f03b" containerName="revoke-keys" Nov 03 20:39:37.632698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.632588 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="3671b5fd-8f95-46af-a40d-35b318d9f03b" containerName="revoke-keys" Nov 03 20:39:37.636598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.636576 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r"] Nov 03 20:39:37.636725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.636685 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.642345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.642325 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:39:37.642472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.642358 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:37.642472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.642415 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:39:37.642472 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.642423 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:39:37.834711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.834677 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpgd5\" (UniqueName: \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.834904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.834730 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.834904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.834790 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.935706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.935622 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpgd5\" (UniqueName: \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.935706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.935677 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.935900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.935794 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.936311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.936286 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.938013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.937988 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.948211 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.948184 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpgd5\" (UniqueName: \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\") pod \"maas-api-revoke-sub-ae594979-wz76r\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:37.951960 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:37.951944 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:38.227909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:38.227884 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r"] Nov 03 20:39:38.230167 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:39:38.230100 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0a79f6_8ced_4f56_b7ec_8d3c0cee143c.slice/crio-bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2 WatchSource:0}: Error finding container bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2: Status 404 returned error can't find the container with id bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2 Nov 03 20:39:38.629369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:38.629334 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" event={"ID":"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c","Type":"ContainerStarted","Data":"122bce20132d60468ed98c2fb4a34f74a19342478c9f0dc3baef09f8261eb4b4"} Nov 03 20:39:38.629582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:38.629390 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" event={"ID":"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c","Type":"ContainerStarted","Data":"bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2"} Nov 03 20:39:38.648950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:38.648900 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" podStartSLOduration=1.64888714 podStartE2EDuration="1.64888714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:38.646661755 +0000 UTC m=+2519.371669350" watchObservedRunningTime="2026-11-03 20:39:38.64888714 +0000 UTC m=+2519.373894735" Nov 03 20:39:39.637215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:39.637180 5954 generic.go:356] "Generic (PLEG): container finished" podID="dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" containerID="122bce20132d60468ed98c2fb4a34f74a19342478c9f0dc3baef09f8261eb4b4" exitCode=0 Nov 03 20:39:39.637396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:39.637261 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" event={"ID":"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c","Type":"ContainerDied","Data":"122bce20132d60468ed98c2fb4a34f74a19342478c9f0dc3baef09f8261eb4b4"} Nov 03 20:39:40.773628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.773604 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:40.863707 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.863674 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\" (UniqueName: \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\") pod \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " Nov 03 20:39:40.863878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.863784 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\") pod \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " Nov 03 20:39:40.863878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.863818 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\") pod \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\" (UID: \"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c\") " Nov 03 20:39:40.864095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.864074 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca" pod "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" (UID: "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:40.865827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.865803 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca" pod "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" (UID: "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:40.865962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.865882 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5" pod "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" (UID: "dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c"). InnerVolumeSpecName "kube-api-access-gpgd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:40.965426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.965357 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpgd5\" (UniqueName: \"kubernetes.io/projected/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-kube-api-access-gpgd5\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:40.965426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.965380 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:40.965426 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:40.965391 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.651779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:41.651751 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" Nov 03 20:39:41.651971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:41.651789 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r" event={"ID":"dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c","Type":"ContainerDied","Data":"bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2"} Nov 03 20:39:41.651971 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:41.651861 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf778a75792168502ef6f52bb77bff1e84571ccefaea9b41343704bd53f53bb2" Nov 03 20:39:47.540857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:47.540826 5954 scope.go:122] "RemoveContainer" containerID="55c0d897bbfe08a97009bc52265ee20e2d96584f6d5b60facee2c5db2187fd55" Nov 03 20:39:48.556234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:48.556204 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:39:49.163889 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:49.163859 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:53.023938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:53.023858 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85"] Nov 03 20:39:53.028871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:53.028837 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7cb93d-zqq85"] Nov 03 20:39:53.403785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:53.403753 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6da2f3f-0bb1-40db-a0cc-01f667876fa2" path="/var/lib/kubelet/pods/b6da2f3f-0bb1-40db-a0cc-01f667876fa2/volumes" Nov 03 20:39:58.526620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:39:58.526588 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:00.258431 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:40:00.258391 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:40:01.025235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:01.025203 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6"] Nov 03 20:40:01.028269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:01.028239 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b76dc6-gbng6"] Nov 03 20:40:01.402970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:01.402941 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc306b2-c710-4a5d-9185-1b396131a8fa" path="/var/lib/kubelet/pods/3fc306b2-c710-4a5d-9185-1b396131a8fa/volumes" Nov 03 20:40:01.561061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:01.561027 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:05.344854 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.344819 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz"] Nov 03 20:40:05.345289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.345275 5954 cpu_manager.go:414] "containerMap: removing container" podUID="dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" containerName="revoke-keys" Nov 03 20:40:05.345340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.345291 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" containerName="revoke-keys" Nov 03 20:40:05.345395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.345385 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" containerName="revoke-keys" Nov 03 20:40:05.350475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.350453 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.356771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.356745 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:05.356771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.356752 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:40:05.356999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.356771 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:40:05.358007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.357986 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:40:05.367222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.367203 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz"] Nov 03 20:40:05.435705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.435655 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.435886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.435750 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.435886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.435778 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtw8\" (UniqueName: \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.537225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.537174 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.537225 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.537225 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtw8\" (UniqueName: \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.537505 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.537403 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.537841 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.537818 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.539694 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.539673 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.549644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.549622 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtw8\" (UniqueName: \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\") pod \"maas-api-revoke-sub-a4aef7d7-vhwqz\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.662760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.662724 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:05.939495 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:05.939466 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz"] Nov 03 20:40:05.941223 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:40:05.941191 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f4964e_0584_4cae_9939_0cea56c6c52a.slice/crio-fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd WatchSource:0}: Error finding container fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd: Status 404 returned error can't find the container with id fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd Nov 03 20:40:06.812788 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:06.812750 5954 generic.go:356] "Generic (PLEG): container finished" podID="02f4964e-0584-4cae-9939-0cea56c6c52a" containerID="d94162fd5c8c3554d2e44bf8801582465613bfb0cc1d0faa3bec2eda0519b3b2" exitCode=0 Nov 03 20:40:06.813240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:06.812837 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" event={"ID":"02f4964e-0584-4cae-9939-0cea56c6c52a","Type":"ContainerDied","Data":"d94162fd5c8c3554d2e44bf8801582465613bfb0cc1d0faa3bec2eda0519b3b2"} Nov 03 20:40:06.813240 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:06.812910 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" event={"ID":"02f4964e-0584-4cae-9939-0cea56c6c52a","Type":"ContainerStarted","Data":"fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd"} Nov 03 20:40:07.956560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:07.956532 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:08.060159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.060099 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\" (UniqueName: \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\") pod \"02f4964e-0584-4cae-9939-0cea56c6c52a\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " Nov 03 20:40:08.060335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.060182 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\") pod \"02f4964e-0584-4cae-9939-0cea56c6c52a\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " Nov 03 20:40:08.060335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.060303 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\") pod \"02f4964e-0584-4cae-9939-0cea56c6c52a\" (UID: \"02f4964e-0584-4cae-9939-0cea56c6c52a\") " Nov 03 20:40:08.060569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.060544 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca" pod "02f4964e-0584-4cae-9939-0cea56c6c52a" (UID: "02f4964e-0584-4cae-9939-0cea56c6c52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:08.062174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.062152 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca" pod "02f4964e-0584-4cae-9939-0cea56c6c52a" (UID: "02f4964e-0584-4cae-9939-0cea56c6c52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:08.062320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.062303 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8" pod "02f4964e-0584-4cae-9939-0cea56c6c52a" (UID: "02f4964e-0584-4cae-9939-0cea56c6c52a"). InnerVolumeSpecName "kube-api-access-mxtw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:08.161800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.161770 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:08.161800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.161794 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtw8\" (UniqueName: \"kubernetes.io/projected/02f4964e-0584-4cae-9939-0cea56c6c52a-kube-api-access-mxtw8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:08.161800 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.161805 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f4964e-0584-4cae-9939-0cea56c6c52a-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:08.825098 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.825069 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" Nov 03 20:40:08.825383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.825133 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz" event={"ID":"02f4964e-0584-4cae-9939-0cea56c6c52a","Type":"ContainerDied","Data":"fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd"} Nov 03 20:40:08.825383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:08.825183 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fefa436a241d556455d8d41ea3b3a7d56b04e7eb42ef7306e14465eff4382ecd" Nov 03 20:40:11.025089 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.025053 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx"] Nov 03 20:40:11.029682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.029654 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9"] Nov 03 20:40:11.031824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.031798 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aae205c-l6rgx"] Nov 03 20:40:11.033532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.033514 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4adfaf-qs7g9"] Nov 03 20:40:11.403639 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.403601 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9fc097-cec8-4a6b-9fe6-0841ef43c582" path="/var/lib/kubelet/pods/7a9fc097-cec8-4a6b-9fe6-0841ef43c582/volumes" Nov 03 20:40:11.404095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:11.404073 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f682e1-e5e5-402b-a568-c7bed86c6ee2" path="/var/lib/kubelet/pods/f3f682e1-e5e5-402b-a568-c7bed86c6ee2/volumes" Nov 03 20:40:15.033036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:15.033001 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr"] Nov 03 20:40:15.036457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:15.036432 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12b4a4d7-snkpr"] Nov 03 20:40:15.403164 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:15.403125 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295784c0-89db-405f-b53d-3533f77b8098" path="/var/lib/kubelet/pods/295784c0-89db-405f-b53d-3533f77b8098/volumes" Nov 03 20:40:16.023321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:16.023286 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt"] Nov 03 20:40:16.027897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:16.027870 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e908b41b-bsmqt"] Nov 03 20:40:16.263311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:16.263273 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:17.404793 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:17.404762 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13687593-3523-4ed6-8770-43766068baa6" path="/var/lib/kubelet/pods/13687593-3523-4ed6-8770-43766068baa6/volumes" Nov 03 20:40:21.403292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:21.403263 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:24.247270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.247235 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p"] Nov 03 20:40:24.247833 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.247814 5954 cpu_manager.go:414] "containerMap: removing container" podUID="02f4964e-0584-4cae-9939-0cea56c6c52a" containerName="revoke-keys" Nov 03 20:40:24.247883 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.247835 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f4964e-0584-4cae-9939-0cea56c6c52a" containerName="revoke-keys" Nov 03 20:40:24.247941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.247932 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f4964e-0584-4cae-9939-0cea56c6c52a" containerName="revoke-keys" Nov 03 20:40:24.252667 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.252646 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.258273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.258250 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:24.259508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.259483 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:40:24.259666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.259524 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:40:24.259666 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.259660 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:40:24.262917 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.262893 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p"] Nov 03 20:40:24.329614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.329583 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.329787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.329639 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.329787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.329760 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgdq\" (UniqueName: \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.439021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.432639 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.439021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.432755 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.439021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.432828 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgdq\" (UniqueName: \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.439021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.433930 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.439628 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.439604 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.445168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.445143 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgdq\" (UniqueName: \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\") pod \"maas-api-revoke-sub-79805ca9-6v57p\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.568050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.567944 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:24.845845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.845820 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p"] Nov 03 20:40:24.847881 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:40:24.847851 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6f29b4_db8a_4e01_a0f8_1a4c73bf6a1c.slice/crio-08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35 WatchSource:0}: Error finding container 08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35: Status 404 returned error can't find the container with id 08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35 Nov 03 20:40:24.921798 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:24.921764 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" event={"ID":"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c","Type":"ContainerStarted","Data":"08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35"} Nov 03 20:40:25.929367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:25.929279 5954 generic.go:356] "Generic (PLEG): container finished" podID="2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" containerID="9892d10d216757c096393e913012ed6a677340d6f92800c04de952fa5fb55acb" exitCode=0 Nov 03 20:40:25.929367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:25.929315 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" event={"ID":"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c","Type":"ContainerDied","Data":"9892d10d216757c096393e913012ed6a677340d6f92800c04de952fa5fb55acb"} Nov 03 20:40:27.077745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.077718 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:27.159450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.159419 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\") pod \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " Nov 03 20:40:27.159631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.159613 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\" (UniqueName: \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\") pod \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " Nov 03 20:40:27.159709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.159693 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\") pod \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\" (UID: \"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c\") " Nov 03 20:40:27.160046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.160017 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca" pod "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" (UID: "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:27.161519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.161496 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca" pod "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" (UID: "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:27.161614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.161596 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq" pod "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" (UID: "2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c"). InnerVolumeSpecName "kube-api-access-pfgdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:27.260785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.260699 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:27.260785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.260736 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:27.260785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.260750 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgdq\" (UniqueName: \"kubernetes.io/projected/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c-kube-api-access-pfgdq\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:27.942737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.942697 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" event={"ID":"2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c","Type":"ContainerDied","Data":"08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35"} Nov 03 20:40:27.942925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.942761 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08326449e315785b6e68dcea5187a2a6dc146d3877251cd79eec56f21829af35" Nov 03 20:40:27.942925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:27.942713 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p" Nov 03 20:40:30.257639 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:40:30.257597 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:40:35.232601 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:35.232565 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:39.132322 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:39.132286 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:42.716807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.716767 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7"] Nov 03 20:40:42.717520 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.717497 5954 cpu_manager.go:414] "containerMap: removing container" podUID="2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" containerName="revoke-keys" Nov 03 20:40:42.717572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.717523 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" containerName="revoke-keys" Nov 03 20:40:42.717701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.717688 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" containerName="revoke-keys" Nov 03 20:40:42.721432 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.721415 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.726241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.726096 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:40:42.726241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.726211 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:40:42.726569 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.726548 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:42.726670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.726662 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:40:42.729970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.729938 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7"] Nov 03 20:40:42.837731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.837694 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.837922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.837740 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79clc\" (UniqueName: \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.837922 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.837817 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.938398 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.938367 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.938611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.938413 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79clc\" (UniqueName: \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.938611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.938462 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.939076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.939051 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.940925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.940905 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:42.951037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:42.951017 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79clc\" (UniqueName: \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\") pod \"maas-api-revoke-sub-0b7e8edb-fgbx7\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:43.039258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:43.039175 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:43.317734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:43.317699 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7"] Nov 03 20:40:43.319261 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:40:43.319235 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eec132a_c614_4af7_bd35_ddf9a738ff2e.slice/crio-248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b WatchSource:0}: Error finding container 248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b: Status 404 returned error can't find the container with id 248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b Nov 03 20:40:44.041506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:44.041387 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" event={"ID":"0eec132a-c614-4af7-bd35-ddf9a738ff2e","Type":"ContainerStarted","Data":"4744a9474fe06c1ea02c541371c712f1150d97eb7c2c9258c4bab0f35304e939"} Nov 03 20:40:44.041506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:44.041473 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" event={"ID":"0eec132a-c614-4af7-bd35-ddf9a738ff2e","Type":"ContainerStarted","Data":"248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b"} Nov 03 20:40:44.062897 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:44.062850 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" podStartSLOduration=2.062837766 podStartE2EDuration="2.062837766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:44.061258987 +0000 UTC m=+2584.786266594" watchObservedRunningTime="2026-11-03 20:40:44.062837766 +0000 UTC m=+2584.787845386" Nov 03 20:40:45.049230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:45.049192 5954 generic.go:356] "Generic (PLEG): container finished" podID="0eec132a-c614-4af7-bd35-ddf9a738ff2e" containerID="4744a9474fe06c1ea02c541371c712f1150d97eb7c2c9258c4bab0f35304e939" exitCode=0 Nov 03 20:40:45.049622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:45.049241 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" event={"ID":"0eec132a-c614-4af7-bd35-ddf9a738ff2e","Type":"ContainerDied","Data":"4744a9474fe06c1ea02c541371c712f1150d97eb7c2c9258c4bab0f35304e939"} Nov 03 20:40:46.188835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.188811 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:46.270940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.270909 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\") pod \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " Nov 03 20:40:46.271161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.270953 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\") pod \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " Nov 03 20:40:46.271161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.271027 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\" (UniqueName: \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\") pod \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\" (UID: \"0eec132a-c614-4af7-bd35-ddf9a738ff2e\") " Nov 03 20:40:46.271373 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.271348 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca" pod "0eec132a-c614-4af7-bd35-ddf9a738ff2e" (UID: "0eec132a-c614-4af7-bd35-ddf9a738ff2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:46.273205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.273177 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc" pod "0eec132a-c614-4af7-bd35-ddf9a738ff2e" (UID: "0eec132a-c614-4af7-bd35-ddf9a738ff2e"). InnerVolumeSpecName "kube-api-access-79clc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:46.273316 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.273216 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca" pod "0eec132a-c614-4af7-bd35-ddf9a738ff2e" (UID: "0eec132a-c614-4af7-bd35-ddf9a738ff2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:46.372609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.372582 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:46.372609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.372609 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eec132a-c614-4af7-bd35-ddf9a738ff2e-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:46.372807 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:46.372621 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79clc\" (UniqueName: \"kubernetes.io/projected/0eec132a-c614-4af7-bd35-ddf9a738ff2e-kube-api-access-79clc\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:47.063259 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.063226 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" Nov 03 20:40:47.063457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.063268 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7" event={"ID":"0eec132a-c614-4af7-bd35-ddf9a738ff2e","Type":"ContainerDied","Data":"248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b"} Nov 03 20:40:47.063457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.063327 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="248944c4045fe828f43832691cad239ef2b8e4871b71aa9dae4d79667f3c8b6b" Nov 03 20:40:47.565461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.565428 5954 scope.go:122] "RemoveContainer" containerID="672ef62045193370d3a341e037cc79f907aca3b8752a219b644f74b7ca597d8b" Nov 03 20:40:47.574677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.574658 5954 scope.go:122] "RemoveContainer" containerID="d044a8639a6475613847ee60345ad78e32b43b217047edbb02923c97dd526291" Nov 03 20:40:47.583757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:47.583739 5954 scope.go:122] "RemoveContainer" containerID="4b60f35ff90031fb3695aeaf42522e5d5f7b31ea51763f542fe59f7b85fdb574" Nov 03 20:40:49.221043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:49.221008 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:53.534382 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.534329 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj"] Nov 03 20:40:53.535155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.535127 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0eec132a-c614-4af7-bd35-ddf9a738ff2e" containerName="revoke-keys" Nov 03 20:40:53.535155 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.535153 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eec132a-c614-4af7-bd35-ddf9a738ff2e" containerName="revoke-keys" Nov 03 20:40:53.535393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.535377 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eec132a-c614-4af7-bd35-ddf9a738ff2e" containerName="revoke-keys" Nov 03 20:40:53.541201 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.541180 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.545817 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.545791 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:53.546035 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.545875 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj"] Nov 03 20:40:53.546291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.546272 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:40:53.546368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.546312 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:40:53.546469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.546455 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:40:53.649093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.649056 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmn6t\" (UniqueName: \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.649298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.649215 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.649344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.649321 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.731900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.731862 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:40:53.750771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.750739 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmn6t\" (UniqueName: \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.750949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.750801 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.750949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.750858 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.751420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.751400 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.753081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.753055 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.763625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.763599 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmn6t\" (UniqueName: \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\") pod \"maas-api-revoke-sub-3d4156f1-vv6qj\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:53.859698 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:53.859668 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:54.139955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:54.139924 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj"] Nov 03 20:40:54.149747 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:40:54.149682 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42644ab_03aa_4628_90da_a87006de780a.slice/crio-6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786 WatchSource:0}: Error finding container 6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786: Status 404 returned error can't find the container with id 6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786 Nov 03 20:40:55.111372 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:55.111341 5954 generic.go:356] "Generic (PLEG): container finished" podID="f42644ab-03aa-4628-90da-a87006de780a" containerID="21d1d0e080c62a3769a48d46383f0265908e2432ae691e39f8604aed9caf62a1" exitCode=0 Nov 03 20:40:55.111871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:55.111425 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" event={"ID":"f42644ab-03aa-4628-90da-a87006de780a","Type":"ContainerDied","Data":"21d1d0e080c62a3769a48d46383f0265908e2432ae691e39f8604aed9caf62a1"} Nov 03 20:40:55.111871 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:55.111477 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" event={"ID":"f42644ab-03aa-4628-90da-a87006de780a","Type":"ContainerStarted","Data":"6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786"} Nov 03 20:40:56.254777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.254752 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:56.380445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.380365 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\") pod \"f42644ab-03aa-4628-90da-a87006de780a\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " Nov 03 20:40:56.380445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.380438 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\") pod \"f42644ab-03aa-4628-90da-a87006de780a\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " Nov 03 20:40:56.380648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.380465 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\" (UniqueName: \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\") pod \"f42644ab-03aa-4628-90da-a87006de780a\" (UID: \"f42644ab-03aa-4628-90da-a87006de780a\") " Nov 03 20:40:56.380830 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.380799 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca" pod "f42644ab-03aa-4628-90da-a87006de780a" (UID: "f42644ab-03aa-4628-90da-a87006de780a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:56.380952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.380909 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f42644ab-03aa-4628-90da-a87006de780a-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:56.382537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.382507 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca" pod "f42644ab-03aa-4628-90da-a87006de780a" (UID: "f42644ab-03aa-4628-90da-a87006de780a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:56.382620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.382559 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t" pod "f42644ab-03aa-4628-90da-a87006de780a" (UID: "f42644ab-03aa-4628-90da-a87006de780a"). InnerVolumeSpecName "kube-api-access-jmn6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:56.481861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.481829 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f42644ab-03aa-4628-90da-a87006de780a-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:56.481861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:56.481856 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmn6t\" (UniqueName: \"kubernetes.io/projected/f42644ab-03aa-4628-90da-a87006de780a-kube-api-access-jmn6t\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.123976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:57.123946 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" Nov 03 20:40:57.123976 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:57.123956 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj" event={"ID":"f42644ab-03aa-4628-90da-a87006de780a","Type":"ContainerDied","Data":"6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786"} Nov 03 20:40:57.124220 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:40:57.124003 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b9a14dd9e3aa6a08c06cf456158989869b540ebdd907cab9dc23dfe1fc92786" Nov 03 20:41:00.256920 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:41:00.256881 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:41:04.433622 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:04.433591 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:09.562811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:09.562775 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:12.813845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.813808 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj"] Nov 03 20:41:12.820970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.820938 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vv6qj"] Nov 03 20:41:12.840414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.840380 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd"] Nov 03 20:41:12.840840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.840827 5954 cpu_manager.go:414] "containerMap: removing container" podUID="f42644ab-03aa-4628-90da-a87006de780a" containerName="revoke-keys" Nov 03 20:41:12.840886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.840842 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42644ab-03aa-4628-90da-a87006de780a" containerName="revoke-keys" Nov 03 20:41:12.840948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.840939 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42644ab-03aa-4628-90da-a87006de780a" containerName="revoke-keys" Nov 03 20:41:12.844661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.844642 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:12.850131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.850082 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:41:12.851336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.851293 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:12.851463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.851343 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:41:12.851463 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.851355 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:41:12.854290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.854273 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd"] Nov 03 20:41:12.955741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.955710 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:12.955932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.955767 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:12.955932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:12.955880 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6j9\" (UniqueName: \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.056532 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.056494 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.056724 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.056587 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6j9\" (UniqueName: \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.056799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.056714 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.057221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.057192 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.058821 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.058799 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.071723 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.071667 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6j9\" (UniqueName: \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\") pod \"maas-api-revoke-sub-3d4156f1-wwlfd\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.161208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.161174 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:13.403741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.403696 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42644ab-03aa-4628-90da-a87006de780a" path="/var/lib/kubelet/pods/f42644ab-03aa-4628-90da-a87006de780a/volumes" Nov 03 20:41:13.442062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:13.442027 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd"] Nov 03 20:41:13.444326 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:41:13.444300 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a85f47_2d77_4217_8cf6_d64d0b4c4699.slice/crio-b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b WatchSource:0}: Error finding container b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b: Status 404 returned error can't find the container with id b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b Nov 03 20:41:14.245826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:14.245799 5954 generic.go:356] "Generic (PLEG): container finished" podID="98a85f47-2d77-4217-8cf6-d64d0b4c4699" containerID="66585259ca8e9b3cb8da3e1f7de7ab04bfc9bfbfe0ffea7c1f8eaf351cd6923a" exitCode=0 Nov 03 20:41:14.246139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:14.245876 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" event={"ID":"98a85f47-2d77-4217-8cf6-d64d0b4c4699","Type":"ContainerDied","Data":"66585259ca8e9b3cb8da3e1f7de7ab04bfc9bfbfe0ffea7c1f8eaf351cd6923a"} Nov 03 20:41:14.246139 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:14.245929 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" event={"ID":"98a85f47-2d77-4217-8cf6-d64d0b4c4699","Type":"ContainerStarted","Data":"b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b"} Nov 03 20:41:15.389073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.389050 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:15.482882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.482848 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\" (UniqueName: \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\") pod \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " Nov 03 20:41:15.483060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.482940 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\") pod \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " Nov 03 20:41:15.483060 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.483033 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\") pod \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\" (UID: \"98a85f47-2d77-4217-8cf6-d64d0b4c4699\") " Nov 03 20:41:15.483436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.483407 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca" pod "98a85f47-2d77-4217-8cf6-d64d0b4c4699" (UID: "98a85f47-2d77-4217-8cf6-d64d0b4c4699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:15.485017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.484990 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9" pod "98a85f47-2d77-4217-8cf6-d64d0b4c4699" (UID: "98a85f47-2d77-4217-8cf6-d64d0b4c4699"). InnerVolumeSpecName "kube-api-access-xv6j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:15.485202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.485033 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca" pod "98a85f47-2d77-4217-8cf6-d64d0b4c4699" (UID: "98a85f47-2d77-4217-8cf6-d64d0b4c4699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:15.584589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.584505 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:15.584589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.584540 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6j9\" (UniqueName: \"kubernetes.io/projected/98a85f47-2d77-4217-8cf6-d64d0b4c4699-kube-api-access-xv6j9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:15.584589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:15.584551 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a85f47-2d77-4217-8cf6-d64d0b4c4699-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.263614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:16.263580 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" Nov 03 20:41:16.263818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:16.263587 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd" event={"ID":"98a85f47-2d77-4217-8cf6-d64d0b4c4699","Type":"ContainerDied","Data":"b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b"} Nov 03 20:41:16.263818 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:16.263697 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81b26c35c250ebd443abf22eb878d804c1c46e2d635db8bc8c277ad57eae39b" Nov 03 20:41:21.025263 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.025156 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4"] Nov 03 20:41:21.027352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.027311 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g"] Nov 03 20:41:21.029534 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.029504 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d828537-99c8g"] Nov 03 20:41:21.031206 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.031179 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b814e8f1-r87m4"] Nov 03 20:41:21.403708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.403677 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa850e2-42d4-44f6-8db7-cffb2d70d54b" path="/var/lib/kubelet/pods/3aa850e2-42d4-44f6-8db7-cffb2d70d54b/volumes" Nov 03 20:41:21.404103 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:21.404089 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98907a5f-88f4-488c-8932-dcec6dabc6bf" path="/var/lib/kubelet/pods/98907a5f-88f4-488c-8932-dcec6dabc6bf/volumes" Nov 03 20:41:23.748508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:23.748473 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:27.058557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:27.058517 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:30.257123 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:41:30.257074 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:41:34.323442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:34.323405 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:41.642416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.642384 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g"] Nov 03 20:41:41.642835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.642803 5954 cpu_manager.go:414] "containerMap: removing container" podUID="98a85f47-2d77-4217-8cf6-d64d0b4c4699" containerName="revoke-keys" Nov 03 20:41:41.642835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.642814 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a85f47-2d77-4217-8cf6-d64d0b4c4699" containerName="revoke-keys" Nov 03 20:41:41.642918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.642908 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a85f47-2d77-4217-8cf6-d64d0b4c4699" containerName="revoke-keys" Nov 03 20:41:41.646054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.646038 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.649699 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.649676 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:41:41.650660 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.650644 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:41.650750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.650718 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:41:41.651023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.651007 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:41:41.655750 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.655724 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g"] Nov 03 20:41:41.743202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.743161 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jjk\" (UniqueName: \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.743395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.743236 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.743395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.743369 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.844292 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.844246 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.844471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.844326 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.844471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.844390 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jjk\" (UniqueName: \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.845043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.845022 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.846636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.846616 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.858683 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.858660 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jjk\" (UniqueName: \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\") pod \"maas-api-revoke-sub-e2fc9c5b-88t5g\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:41.960780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:41.960696 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:42.237791 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:42.237766 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g"] Nov 03 20:41:42.239490 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:41:42.239460 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3f43e6_2112_4916_bde0_0cbddb5a77fa.slice/crio-6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd WatchSource:0}: Error finding container 6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd: Status 404 returned error can't find the container with id 6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd Nov 03 20:41:42.430284 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:42.430248 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" event={"ID":"ae3f43e6-2112-4916-bde0-0cbddb5a77fa","Type":"ContainerStarted","Data":"6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd"} Nov 03 20:41:43.437385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:43.437352 5954 generic.go:356] "Generic (PLEG): container finished" podID="ae3f43e6-2112-4916-bde0-0cbddb5a77fa" containerID="61d3db523424d39990db6de68ff7b63b9b404575d1da805101340295b655f994" exitCode=0 Nov 03 20:41:43.437801 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:43.437446 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" event={"ID":"ae3f43e6-2112-4916-bde0-0cbddb5a77fa","Type":"ContainerDied","Data":"61d3db523424d39990db6de68ff7b63b9b404575d1da805101340295b655f994"} Nov 03 20:41:44.585771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.585745 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:44.674117 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.674069 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\") pod \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " Nov 03 20:41:44.674299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.674177 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\") pod \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " Nov 03 20:41:44.674299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.674226 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\" (UniqueName: \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\") pod \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\" (UID: \"ae3f43e6-2112-4916-bde0-0cbddb5a77fa\") " Nov 03 20:41:44.674543 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.674515 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca" pod "ae3f43e6-2112-4916-bde0-0cbddb5a77fa" (UID: "ae3f43e6-2112-4916-bde0-0cbddb5a77fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:44.676154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.676130 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca" pod "ae3f43e6-2112-4916-bde0-0cbddb5a77fa" (UID: "ae3f43e6-2112-4916-bde0-0cbddb5a77fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:44.676384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.676362 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk" pod "ae3f43e6-2112-4916-bde0-0cbddb5a77fa" (UID: "ae3f43e6-2112-4916-bde0-0cbddb5a77fa"). InnerVolumeSpecName "kube-api-access-k2jjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:44.775918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.775842 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:44.775918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.775869 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jjk\" (UniqueName: \"kubernetes.io/projected/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-kube-api-access-k2jjk\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:44.775918 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:44.775880 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3f43e6-2112-4916-bde0-0cbddb5a77fa-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:45.450811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:45.450786 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" Nov 03 20:41:45.450978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:45.450824 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g" event={"ID":"ae3f43e6-2112-4916-bde0-0cbddb5a77fa","Type":"ContainerDied","Data":"6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd"} Nov 03 20:41:45.450978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:45.450870 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf29d40b3acfd75b3e46b1b65a30823c3b1d4321421492e2f32dc55ec82cbdd" Nov 03 20:41:47.618623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.618587 5954 scope.go:122] "RemoveContainer" containerID="149d4185b79aff76239f305baac4a61e27968fdf297f6d4ddd2cb3e9147f3b32" Nov 03 20:41:47.628017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.627996 5954 scope.go:122] "RemoveContainer" containerID="337e57a2ac33d121a930d1eacad78290f6b531d6c58c768ce4f5d65eefe36c61" Nov 03 20:41:47.636701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.636680 5954 scope.go:122] "RemoveContainer" containerID="96811fb867558330bdc81eef3f7b982a318fcb9cc21e4f15ae1e8311c94bdae4" Nov 03 20:41:47.645345 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.645316 5954 scope.go:122] "RemoveContainer" containerID="c5a0a90fa87d9416c162ad4639a5fc6505f2621a4d27812e849c52871ab793c2" Nov 03 20:41:47.653379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.653358 5954 scope.go:122] "RemoveContainer" containerID="3caf5e13f922ee31f9711c790522cd959343fc374acd7cbd75597b50894079d0" Nov 03 20:41:47.661160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.661141 5954 scope.go:122] "RemoveContainer" containerID="7432c22e6d122ee63bdfe497e3a976fea9d73079bd5bc6dbd6fe43be98baf671" Nov 03 20:41:47.668909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.668887 5954 scope.go:122] "RemoveContainer" containerID="086ce3227e22a672df28483b6c3fdaee7799f6d2107bb8263c0c131fd165c850" Nov 03 20:41:47.676890 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.676871 5954 scope.go:122] "RemoveContainer" containerID="d637878c68fb123f6942751d5891b6b99f52fb518bdbd364095eafc41126d7f4" Nov 03 20:41:47.685174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:47.685150 5954 scope.go:122] "RemoveContainer" containerID="06be7614a18263fcc4301a0d1b6fc96a15eef26066c86edd4837ac3e29b56a7e" Nov 03 20:41:49.270966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:49.270930 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:51.871389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.871355 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g"] Nov 03 20:41:51.872032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.872009 5954 cpu_manager.go:414] "containerMap: removing container" podUID="ae3f43e6-2112-4916-bde0-0cbddb5a77fa" containerName="revoke-keys" Nov 03 20:41:51.872101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.872034 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3f43e6-2112-4916-bde0-0cbddb5a77fa" containerName="revoke-keys" Nov 03 20:41:51.872161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.872143 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3f43e6-2112-4916-bde0-0cbddb5a77fa" containerName="revoke-keys" Nov 03 20:41:51.877229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.877210 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:51.882307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.882284 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:51.882903 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.882878 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:41:51.886137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.884254 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g"] Nov 03 20:41:51.918973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.918932 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:41:51.919145 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.919020 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:41:51.945044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.945013 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6qv\" (UniqueName: \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:51.945217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.945050 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:51.945273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:51.945245 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.046172 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.046134 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.046385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.046200 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6qv\" (UniqueName: \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.046385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.046218 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.046857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.046837 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.048515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.048489 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.058908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.058882 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6qv\" (UniqueName: \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\") pod \"maas-api-revoke-sub-5a29efb3-g4t4g\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.194403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.194320 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:52.470905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.470828 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g"] Nov 03 20:41:52.474531 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:41:52.474502 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58740a1f_73af_42d0_84c3_7c9da2460927.slice/crio-82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f WatchSource:0}: Error finding container 82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f: Status 404 returned error can't find the container with id 82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f Nov 03 20:41:52.491466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.491434 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" event={"ID":"58740a1f-73af-42d0-84c3-7c9da2460927","Type":"ContainerStarted","Data":"82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f"} Nov 03 20:41:52.666650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:52.666616 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:41:53.500066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:53.500023 5954 generic.go:356] "Generic (PLEG): container finished" podID="58740a1f-73af-42d0-84c3-7c9da2460927" containerID="a82518b04a6411be88d7fa8b1478efa91b24956135b386fb607918752bc897cc" exitCode=0 Nov 03 20:41:53.500496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:53.500136 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" event={"ID":"58740a1f-73af-42d0-84c3-7c9da2460927","Type":"ContainerDied","Data":"a82518b04a6411be88d7fa8b1478efa91b24956135b386fb607918752bc897cc"} Nov 03 20:41:54.892356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:54.892326 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:41:54.998058 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:54.998022 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\") pod \"58740a1f-73af-42d0-84c3-7c9da2460927\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " Nov 03 20:41:54.998250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:54.998064 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\" (UniqueName: \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\") pod \"58740a1f-73af-42d0-84c3-7c9da2460927\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " Nov 03 20:41:54.998250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:54.998139 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\") pod \"58740a1f-73af-42d0-84c3-7c9da2460927\" (UID: \"58740a1f-73af-42d0-84c3-7c9da2460927\") " Nov 03 20:41:54.998653 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:54.998630 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca" pod "58740a1f-73af-42d0-84c3-7c9da2460927" (UID: "58740a1f-73af-42d0-84c3-7c9da2460927"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:55.000673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.000642 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca" pod "58740a1f-73af-42d0-84c3-7c9da2460927" (UID: "58740a1f-73af-42d0-84c3-7c9da2460927"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:55.000912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.000889 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv" pod "58740a1f-73af-42d0-84c3-7c9da2460927" (UID: "58740a1f-73af-42d0-84c3-7c9da2460927"). InnerVolumeSpecName "kube-api-access-bf6qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:55.098951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.098869 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:55.098951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.098894 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6qv\" (UniqueName: \"kubernetes.io/projected/58740a1f-73af-42d0-84c3-7c9da2460927-kube-api-access-bf6qv\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:55.098951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.098904 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58740a1f-73af-42d0-84c3-7c9da2460927-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:55.512159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.512123 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" event={"ID":"58740a1f-73af-42d0-84c3-7c9da2460927","Type":"ContainerDied","Data":"82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f"} Nov 03 20:41:55.512370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.512189 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82110ce2efe00fb368473856832a64beedef9ac0a79d2a80124330578af2270f" Nov 03 20:41:55.512370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:41:55.512132 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g" Nov 03 20:42:00.256862 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:42:00.256824 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:42:02.765433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:02.765395 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:42:07.023182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:07.023144 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc"] Nov 03 20:42:07.025974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:07.025945 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-7jzfc"] Nov 03 20:42:07.403643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:07.403610 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a04fd3-48c9-421b-9574-2acd9461bd76" path="/var/lib/kubelet/pods/64a04fd3-48c9-421b-9574-2acd9461bd76/volumes" Nov 03 20:42:30.257519 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:42:30.257474 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:42:47.750642 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:47.750609 5954 scope.go:122] "RemoveContainer" containerID="49d7991892906598b554347636367fcbceb35ef2847cff588a1f3a43d4a21e78" Nov 03 20:42:47.759530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:47.759510 5954 scope.go:122] "RemoveContainer" containerID="76acfabfc35ea34d2945160405268f158ac3468266434dd691bb6a3a835775ef" Nov 03 20:42:47.767615 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:47.767592 5954 scope.go:122] "RemoveContainer" containerID="bb073bd8d73ea0585f02eb58862e208f43ebab003fbd4d4c43d5be542d50b8ef" Nov 03 20:42:47.775442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:47.775424 5954 scope.go:122] "RemoveContainer" containerID="8932a9abde14c75f3af6222dda52bc214d0ebc46717a6060f046c782d9c3be5e" Nov 03 20:42:49.325991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:49.325957 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:59.480717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:42:59.480660 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:00.257410 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:43:00.257373 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:43:04.022708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:04.022672 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq"] Nov 03 20:43:04.025605 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:04.025576 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mgbfq"] Nov 03 20:43:05.403367 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:05.403334 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bd275d-b440-40d0-a463-8bbe41ba0b13" path="/var/lib/kubelet/pods/84bd275d-b440-40d0-a463-8bbe41ba0b13/volumes" Nov 03 20:43:23.023604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:23.023565 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld"] Nov 03 20:43:23.026882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:23.026852 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lxnld"] Nov 03 20:43:23.404086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:23.404050 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d3f304-ce22-4f3d-bf52-984ed7f61dd4" path="/var/lib/kubelet/pods/b0d3f304-ce22-4f3d-bf52-984ed7f61dd4/volumes" Nov 03 20:43:30.257840 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:43:30.257797 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:43:34.022346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:34.022310 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m"] Nov 03 20:43:34.025334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:34.025308 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h2f5m"] Nov 03 20:43:35.404290 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:35.404258 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd807bc-59a4-4bee-8096-4445150c7999" path="/var/lib/kubelet/pods/6dd807bc-59a4-4bee-8096-4445150c7999/volumes" Nov 03 20:43:47.807809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:47.807774 5954 scope.go:122] "RemoveContainer" containerID="995bbbe4f1d11e3147c31020d1a6742a225eb7b47a22dabde1b4d74de4a4b7ac" Nov 03 20:43:49.354861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:43:49.354826 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:00.258196 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:44:00.258159 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:44:10.023086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:10.023046 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs"] Nov 03 20:44:10.025710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:10.025681 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-djdfs"] Nov 03 20:44:11.403347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:11.403312 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7818f2c0-904d-41aa-8423-cda90a11d150" path="/var/lib/kubelet/pods/7818f2c0-904d-41aa-8423-cda90a11d150/volumes" Nov 03 20:44:15.810334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:15.810293 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:15.910692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:15.910655 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:18.572163 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:18.572131 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:19.411713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:19.411684 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:20.021895 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:20.021799 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5"] Nov 03 20:44:20.031336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:20.025946 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tbtk5"] Nov 03 20:44:21.403583 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:21.403540 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3671b5fd-8f95-46af-a40d-35b318d9f03b" path="/var/lib/kubelet/pods/3671b5fd-8f95-46af-a40d-35b318d9f03b/volumes" Nov 03 20:44:30.257681 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:44:30.257643 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:44:41.025611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:41.025574 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r"] Nov 03 20:44:41.028331 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:41.028307 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-wz76r"] Nov 03 20:44:41.404124 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:41.404076 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c" path="/var/lib/kubelet/pods/dd0a79f6-8ced-4f56-b7ec-8d3c0cee143c/volumes" Nov 03 20:44:47.830725 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:47.830693 5954 scope.go:122] "RemoveContainer" containerID="bf092a3a372c71b7e696b0d5de802b475b0feed92ae302cd538610355e91695e" Nov 03 20:44:47.839584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:47.839561 5954 scope.go:122] "RemoveContainer" containerID="b101841e1d2a3c1ab3362ea1198ad389eaebc572e1ff1806c34dcd979fd199b2" Nov 03 20:44:47.847633 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:47.847611 5954 scope.go:122] "RemoveContainer" containerID="ff1cc1b74019758f87a3149201452d6c3fc63b3a3e6c6f856f8f2274e15dc71e" Nov 03 20:44:49.433002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:44:49.432967 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:00.150202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.150166 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:00.150677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.150660 5954 cpu_manager.go:414] "containerMap: removing container" podUID="58740a1f-73af-42d0-84c3-7c9da2460927" containerName="revoke-keys" Nov 03 20:45:00.150728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.150678 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58740a1f-73af-42d0-84c3-7c9da2460927" containerName="revoke-keys" Nov 03 20:45:00.150780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.150771 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="58740a1f-73af-42d0-84c3-7c9da2460927" containerName="revoke-keys" Nov 03 20:45:00.155369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.155350 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.160053 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.160034 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:45:00.161645 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.161616 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:45:00.161756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.161684 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:45:00.173138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.173099 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:00.227031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.226996 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n9h8\" (UniqueName: \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.227199 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.227163 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.256656 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:45:00.256613 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:45:00.328007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.327976 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n9h8\" (UniqueName: \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.328175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.328099 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.328747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.328727 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.340944 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.340918 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n9h8\" (UniqueName: \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\") pod \"maas-api-key-cleanup-29895645-x4w99\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.468763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.468681 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:00.744215 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:00.744188 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:00.746227 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:45:00.746192 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfaf72f_c69a_4661_a1ca_8423d6f198fb.slice/crio-7e080a651e8492c6036d39aca982df16f521610f9fcd5c442baea69a5e16cf9e WatchSource:0}: Error finding container 7e080a651e8492c6036d39aca982df16f521610f9fcd5c442baea69a5e16cf9e: Status 404 returned error can't find the container with id 7e080a651e8492c6036d39aca982df16f521610f9fcd5c442baea69a5e16cf9e Nov 03 20:45:01.531327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:01.531286 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerStarted","Data":"0472f3acd4ee4e9af44eea0a6e2ac9538f71d92d59751ced5293f68f28229560"} Nov 03 20:45:01.531829 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:01.531348 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerStarted","Data":"7e080a651e8492c6036d39aca982df16f521610f9fcd5c442baea69a5e16cf9e"} Nov 03 20:45:01.551288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:01.551240 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" podStartSLOduration=1.551227068 podStartE2EDuration="1.551227068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:01.549101553 +0000 UTC m=+2842.274109177" watchObservedRunningTime="2026-11-03 20:45:01.551227068 +0000 UTC m=+2842.276234664" Nov 03 20:45:02.538434 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:02.538401 5954 generic.go:356] "Generic (PLEG): container finished" podID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerID="0472f3acd4ee4e9af44eea0a6e2ac9538f71d92d59751ced5293f68f28229560" exitCode=60 Nov 03 20:45:02.538844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:02.538495 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerDied","Data":"0472f3acd4ee4e9af44eea0a6e2ac9538f71d92d59751ced5293f68f28229560"} Nov 03 20:45:02.538844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:02.538748 5954 scope.go:122] "RemoveContainer" containerID="0472f3acd4ee4e9af44eea0a6e2ac9538f71d92d59751ced5293f68f28229560" Nov 03 20:45:03.545388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:03.545351 5954 generic.go:356] "Generic (PLEG): container finished" podID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerID="7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048" exitCode=60 Nov 03 20:45:03.545860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:03.545421 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerDied","Data":"7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048"} Nov 03 20:45:03.545860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:03.545477 5954 scope.go:122] "RemoveContainer" containerID="0472f3acd4ee4e9af44eea0a6e2ac9538f71d92d59751ced5293f68f28229560" Nov 03 20:45:03.545860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:03.545673 5954 scope.go:122] "RemoveContainer" containerID="7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048" Nov 03 20:45:03.546035 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:45:03.545911 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-x4w99_odh-ai-gateway-infra(0dfaf72f-c69a-4661-a1ca-8423d6f198fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" Nov 03 20:45:04.552682 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:04.552654 5954 scope.go:122] "RemoveContainer" containerID="7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048" Nov 03 20:45:04.553209 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:45:04.552968 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-x4w99_odh-ai-gateway-infra(0dfaf72f-c69a-4661-a1ca-8423d6f198fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" Nov 03 20:45:08.023692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:08.023653 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz"] Nov 03 20:45:08.026827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:08.026798 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-vhwqz"] Nov 03 20:45:09.403504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:09.403476 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f4964e-0584-4cae-9939-0cea56c6c52a" path="/var/lib/kubelet/pods/02f4964e-0584-4cae-9939-0cea56c6c52a/volumes" Nov 03 20:45:15.397618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:15.397579 5954 scope.go:122] "RemoveContainer" containerID="7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048" Nov 03 20:45:16.616656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:16.616615 5954 generic.go:356] "Generic (PLEG): container finished" podID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerID="1abc28d8134276a056dfe90762716f11c9573079a67c25aa22803f5629fcec94" exitCode=60 Nov 03 20:45:16.617146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:16.616669 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerDied","Data":"1abc28d8134276a056dfe90762716f11c9573079a67c25aa22803f5629fcec94"} Nov 03 20:45:16.617146 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:16.616773 5954 scope.go:122] "RemoveContainer" containerID="7fd8ee72e563a2b79f14dca8f6a3f51f801e5b3926171293f2f26bd626445048" Nov 03 20:45:16.617251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:16.617215 5954 scope.go:122] "RemoveContainer" containerID="1abc28d8134276a056dfe90762716f11c9573079a67c25aa22803f5629fcec94" Nov 03 20:45:16.617522 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:45:16.617500 5954 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-x4w99_odh-ai-gateway-infra(0dfaf72f-c69a-4661-a1ca-8423d6f198fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" Nov 03 20:45:17.654143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.654098 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:17.796768 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.796743 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:17.910787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.910706 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\" (UniqueName: \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\") pod \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " Nov 03 20:45:17.910938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.910863 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\") pod \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\" (UID: \"0dfaf72f-c69a-4661-a1ca-8423d6f198fb\") " Nov 03 20:45:17.911242 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.911220 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca" pod "0dfaf72f-c69a-4661-a1ca-8423d6f198fb" (UID: "0dfaf72f-c69a-4661-a1ca-8423d6f198fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:17.912789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:17.912771 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8" pod "0dfaf72f-c69a-4661-a1ca-8423d6f198fb" (UID: "0dfaf72f-c69a-4661-a1ca-8423d6f198fb"). InnerVolumeSpecName "kube-api-access-5n9h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:18.012602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.012568 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-maas-api-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.012602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.012596 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n9h8\" (UniqueName: \"kubernetes.io/projected/0dfaf72f-c69a-4661-a1ca-8423d6f198fb-kube-api-access-5n9h8\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.630250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.630209 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" event={"ID":"0dfaf72f-c69a-4661-a1ca-8423d6f198fb","Type":"ContainerDied","Data":"7e080a651e8492c6036d39aca982df16f521610f9fcd5c442baea69a5e16cf9e"} Nov 03 20:45:18.630250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.630249 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99" Nov 03 20:45:18.630488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.630287 5954 scope.go:122] "RemoveContainer" containerID="1abc28d8134276a056dfe90762716f11c9573079a67c25aa22803f5629fcec94" Nov 03 20:45:18.652855 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.652829 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:18.657184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:18.657162 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-x4w99"] Nov 03 20:45:19.404055 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:19.404023 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" path="/var/lib/kubelet/pods/0dfaf72f-c69a-4661-a1ca-8423d6f198fb/volumes" Nov 03 20:45:28.023494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:28.023457 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p"] Nov 03 20:45:28.025882 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:28.025854 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6v57p"] Nov 03 20:45:29.403885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:29.403849 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c" path="/var/lib/kubelet/pods/2b6f29b4-db8a-4e01-a0f8-1a4c73bf6a1c/volumes" Nov 03 20:45:30.257466 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:45:30.257429 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:45:47.022715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.022681 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7"] Nov 03 20:45:47.025506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.025482 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fgbx7"] Nov 03 20:45:47.404018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.403974 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eec132a-c614-4af7-bd35-ddf9a738ff2e" path="/var/lib/kubelet/pods/0eec132a-c614-4af7-bd35-ddf9a738ff2e/volumes" Nov 03 20:45:47.876616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.876587 5954 scope.go:122] "RemoveContainer" containerID="122bce20132d60468ed98c2fb4a34f74a19342478c9f0dc3baef09f8261eb4b4" Nov 03 20:45:47.885360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.885337 5954 scope.go:122] "RemoveContainer" containerID="89d6d5ad560f0c3116685db8ba69f3167197b00b76a33072e3e256562dcc465a" Nov 03 20:45:47.893082 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:47.893062 5954 scope.go:122] "RemoveContainer" containerID="9db30041540bc0c011ab598c1121a2d544e558dc3bf50f6e2db68e48c2809257" Nov 03 20:45:49.544402 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:45:49.544367 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:00.257315 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:46:00.257273 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:46:16.022138 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:16.022086 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd"] Nov 03 20:46:16.024346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:16.024323 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wwlfd"] Nov 03 20:46:17.404210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:17.404178 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a85f47-2d77-4217-8cf6-d64d0b4c4699" path="/var/lib/kubelet/pods/98a85f47-2d77-4217-8cf6-d64d0b4c4699/volumes" Nov 03 20:46:25.045405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:25.045370 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:25.146993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:25.146961 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:30.257271 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:46:30.257229 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:46:45.020526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:45.020493 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g"] Nov 03 20:46:45.023189 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:45.023162 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-88t5g"] Nov 03 20:46:45.403857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:45.403825 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3f43e6-2112-4916-bde0-0cbddb5a77fa" path="/var/lib/kubelet/pods/ae3f43e6-2112-4916-bde0-0cbddb5a77fa/volumes" Nov 03 20:46:47.929891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:47.929862 5954 scope.go:122] "RemoveContainer" containerID="d94162fd5c8c3554d2e44bf8801582465613bfb0cc1d0faa3bec2eda0519b3b2" Nov 03 20:46:47.938777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:47.938750 5954 scope.go:122] "RemoveContainer" containerID="4744a9474fe06c1ea02c541371c712f1150d97eb7c2c9258c4bab0f35304e939" Nov 03 20:46:47.946679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:47.946659 5954 scope.go:122] "RemoveContainer" containerID="9892d10d216757c096393e913012ed6a677340d6f92800c04de952fa5fb55acb" Nov 03 20:46:49.551853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:49.551816 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:55.037007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:55.036968 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g"] Nov 03 20:46:55.038879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:55.038855 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g4t4g"] Nov 03 20:46:55.403445 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:46:55.403413 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58740a1f-73af-42d0-84c3-7c9da2460927" path="/var/lib/kubelet/pods/58740a1f-73af-42d0-84c3-7c9da2460927/volumes" Nov 03 20:47:00.256885 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:47:00.256850 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:47:10.795592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.795555 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5"] Nov 03 20:47:10.796013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.795997 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796011 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796052 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796058 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796134 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796145 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.796218 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.796152 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:10.799326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.799306 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:10.803221 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.803197 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:47:10.803959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.803940 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:47:10.817314 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.817293 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5"] Nov 03 20:47:10.845032 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.845005 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:10.845182 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.845049 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:47:10.927420 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.927389 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:10.927568 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.927436 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:10.927623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:10.927570 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrkm\" (UniqueName: \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.028686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.028654 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.028853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.028721 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.028853 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.028787 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrkm\" (UniqueName: \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.029448 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.029421 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.031009 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.030987 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.040564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.040545 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrkm\" (UniqueName: \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\") pod \"maas-api-revoke-sub-3d4156f1-qr7m5\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.112993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.112970 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:11.387718 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:11.387694 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5"] Nov 03 20:47:11.390016 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:47:11.389989 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dbac80_b4bd_45d2_b411_b56f3adfc55e.slice/crio-74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759 WatchSource:0}: Error finding container 74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759: Status 404 returned error can't find the container with id 74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759 Nov 03 20:47:12.216152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:12.216115 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" event={"ID":"c9dbac80-b4bd-45d2-b411-b56f3adfc55e","Type":"ContainerStarted","Data":"4ca75c20a15977cac03282ccdd02c530b9e11fdd43c237b5a3a138c7461912dc"} Nov 03 20:47:12.216563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:12.216185 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" event={"ID":"c9dbac80-b4bd-45d2-b411-b56f3adfc55e","Type":"ContainerStarted","Data":"74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759"} Nov 03 20:47:12.236476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:12.236426 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" podStartSLOduration=2.236410538 podStartE2EDuration="2.236410538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:12.234019443 +0000 UTC m=+2972.959027039" watchObservedRunningTime="2026-11-03 20:47:12.236410538 +0000 UTC m=+2972.961418133" Nov 03 20:47:13.227040 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:13.227000 5954 generic.go:356] "Generic (PLEG): container finished" podID="c9dbac80-b4bd-45d2-b411-b56f3adfc55e" containerID="4ca75c20a15977cac03282ccdd02c530b9e11fdd43c237b5a3a138c7461912dc" exitCode=0 Nov 03 20:47:13.227489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:13.227091 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" event={"ID":"c9dbac80-b4bd-45d2-b411-b56f3adfc55e","Type":"ContainerDied","Data":"4ca75c20a15977cac03282ccdd02c530b9e11fdd43c237b5a3a138c7461912dc"} Nov 03 20:47:14.374614 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.374582 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:14.460966 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.460935 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\") pod \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " Nov 03 20:47:14.461161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.461004 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\" (UniqueName: \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\") pod \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " Nov 03 20:47:14.461161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.461044 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\") pod \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\" (UID: \"c9dbac80-b4bd-45d2-b411-b56f3adfc55e\") " Nov 03 20:47:14.461386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.461364 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca" pod "c9dbac80-b4bd-45d2-b411-b56f3adfc55e" (UID: "c9dbac80-b4bd-45d2-b411-b56f3adfc55e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:14.461456 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.461440 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:14.463203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.463178 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca" pod "c9dbac80-b4bd-45d2-b411-b56f3adfc55e" (UID: "c9dbac80-b4bd-45d2-b411-b56f3adfc55e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:14.463371 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.463216 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm" pod "c9dbac80-b4bd-45d2-b411-b56f3adfc55e" (UID: "c9dbac80-b4bd-45d2-b411-b56f3adfc55e"). InnerVolumeSpecName "kube-api-access-nqrkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:14.562222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.562093 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrkm\" (UniqueName: \"kubernetes.io/projected/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-kube-api-access-nqrkm\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:14.562222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:14.562138 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9dbac80-b4bd-45d2-b411-b56f3adfc55e-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:15.237827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:15.237796 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" Nov 03 20:47:15.237827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:15.237811 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5" event={"ID":"c9dbac80-b4bd-45d2-b411-b56f3adfc55e","Type":"ContainerDied","Data":"74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759"} Nov 03 20:47:15.238052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:15.237863 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74eab57fbf9f01814f7fb0a983d1795bfc2012e79511b51c7ea9b379f8c10759" Nov 03 20:47:21.659731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:21.659693 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:47:25.656955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:25.656909 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:47:29.254317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:29.254278 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:29.622487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:29.622446 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:29.724656 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:29.724621 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:30.257892 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:47:30.257850 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:47:33.339212 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:33.339176 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:33.447795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:33.447761 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.506022 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.505992 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:47:34.506529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.506461 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:34.506529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.506474 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfaf72f-c69a-4661-a1ca-8423d6f198fb" containerName="cleanup" Nov 03 20:47:34.506529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.506519 5954 cpu_manager.go:414] "containerMap: removing container" podUID="c9dbac80-b4bd-45d2-b411-b56f3adfc55e" containerName="revoke-keys" Nov 03 20:47:34.506529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.506525 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9dbac80-b4bd-45d2-b411-b56f3adfc55e" containerName="revoke-keys" Nov 03 20:47:34.506760 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.506594 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9dbac80-b4bd-45d2-b411-b56f3adfc55e" containerName="revoke-keys" Nov 03 20:47:34.510319 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.510293 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.518247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.518222 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:47:34.524573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.524546 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e95cf7-serving-cert\"" Nov 03 20:47:34.525069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.524932 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:47:34.526271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.525011 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:47:34.526271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.526060 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:47:34.526271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.526233 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:47:34.558451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.558420 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.595769 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.595740 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hf7\" (UniqueName: \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.595965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.595889 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.595965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.595943 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.596071 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.596000 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.674654 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.674583 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.696513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.696485 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.696692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.696570 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.696692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.696613 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hf7\" (UniqueName: \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.696819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.696744 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.697383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.697354 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.698881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.698853 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.699029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.699008 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.708831 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.708807 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hf7\" (UniqueName: \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\") pod \"maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:34.783714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.783615 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.828988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:34.828956 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:35.108072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:35.108048 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:47:35.110257 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:47:35.110223 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366c2341_ea18_459e_a3c5_da9aaeb53155.slice/crio-14f8e65b4a652dc976160dd6764e40cd2bd63d9b5cb638f6936f6cc253016b17 WatchSource:0}: Error finding container 14f8e65b4a652dc976160dd6764e40cd2bd63d9b5cb638f6936f6cc253016b17: Status 404 returned error can't find the container with id 14f8e65b4a652dc976160dd6764e40cd2bd63d9b5cb638f6936f6cc253016b17 Nov 03 20:47:35.113251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:35.113217 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:47:35.355962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:35.355921 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" event={"ID":"366c2341-ea18-459e-a3c5-da9aaeb53155","Type":"ContainerStarted","Data":"14f8e65b4a652dc976160dd6764e40cd2bd63d9b5cb638f6936f6cc253016b17"} Nov 03 20:47:35.503039 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:35.502949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.212033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:36.211994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.362952 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:36.362914 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" event={"ID":"366c2341-ea18-459e-a3c5-da9aaeb53155","Type":"ContainerStarted","Data":"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91"} Nov 03 20:47:36.363175 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:36.363050 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:36.383710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:36.383656 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" podStartSLOduration=1.861659301 podStartE2EDuration="2.3836428s" totalImagesPullingTime="521.983499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:36.382375719 +0000 UTC m=+2997.107383316" watchObservedRunningTime="2026-11-03 20:47:36.3836428 +0000 UTC m=+2997.108650423" Nov 03 20:47:36.948101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:36.948067 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:37.125799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:37.125761 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:47:37.767030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:37.766991 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:38.454549 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:38.454499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.182467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:39.182424 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.850170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:39.850137 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:40.782321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:40.782281 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:41.454186 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:41.454147 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.149687 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:42.149649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.374027 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:42.373994 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:47:42.899308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:42.899275 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:43.654644 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:43.654612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.530230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:44.530196 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.246842 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:45.246806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.053825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:46.053794 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.844616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:46.844571 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.552085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:47.552051 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.978235 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:47.978202 5954 scope.go:122] "RemoveContainer" containerID="21d1d0e080c62a3769a48d46383f0265908e2432ae691e39f8604aed9caf62a1" Nov 03 20:47:47.986671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:47.986645 5954 scope.go:122] "RemoveContainer" containerID="66585259ca8e9b3cb8da3e1f7de7ab04bfc9bfbfe0ffea7c1f8eaf351cd6923a" Nov 03 20:47:47.994366 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:47.994339 5954 scope.go:122] "RemoveContainer" containerID="61d3db523424d39990db6de68ff7b63b9b404575d1da805101340295b655f994" Nov 03 20:47:48.246678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:48.246584 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:48.394269 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:48.394236 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:48.953131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:48.953062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.125665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:49.125630 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.275072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:49.274945 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.709077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:49.709043 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.909574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:49.909495 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:50.283778 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:50.283742 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:50.555273 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:50.555193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.244578 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:51.244546 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.150288 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:52.150253 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.446522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:52.446436 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:47:52.972134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:52.972088 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:53.849834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:53.849793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:54.549557 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:54.549525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.340564 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:55.340525 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:56.089021 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:56.088981 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:56.745471 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:56.745434 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:57.438594 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:57.438563 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:58.133519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:58.133484 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:58.846385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:58.846355 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:59.550160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:47:59.550095 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:00.257912 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:48:00.257876 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:48:00.341247 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:00.341210 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.051002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:02.050962 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.446161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:02.446090 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.248086 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:03.248047 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.950283 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:03.950251 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:04.648224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:04.648188 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:05.436915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:05.436881 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.140878 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:06.140826 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.945542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:06.945508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.657536 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:07.657499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:08.501474 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:08.501414 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:09.047476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:09.047443 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:48:09.388462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:09.388429 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:10.058527 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:10.058493 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:10.753780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:10.753743 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:11.497070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:11.497037 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:12.152389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:12.152359 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:12.839934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:12.839900 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:13.545876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:13.545824 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:14.237143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:14.237096 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:14.933748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:14.933705 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:15.735806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:15.735769 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:16.544600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:16.544563 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:17.236795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:17.236760 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:18.000484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:18.000177 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:18.693469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:18.693429 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:19.339341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:19.339306 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.046001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:20.045968 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.750770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:20.750720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:21.447219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:21.447161 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:22.144590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:22.144557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:22.847826 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:22.847789 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.552102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:23.552069 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:24.246514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:24.246478 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:24.932815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:24.932777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:25.734514 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:25.734481 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:26.436758 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:26.436722 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:27.140796 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:27.140767 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:27.840958 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:27.840927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:28.641574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:28.641540 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:29.343672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:29.343623 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:30.043325 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:30.043286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:30.257746 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:48:30.257713 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:48:30.815713 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:30.815678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:31.445808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:31.445772 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:31.545706 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:31.545675 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:32.319900 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:32.319871 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:32.465851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:32.465821 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:48:33.149850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:33.149813 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:33.952728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:33.952685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.662528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:34.662498 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:35.356299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:35.356265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:36.051561 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:36.051526 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:36.588965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:36.588932 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:36.826665 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:36.826631 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:37.445709 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:37.445682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.247357 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:38.247323 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.948864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:38.948832 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.631488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:39.631457 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:40.432649 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:40.432610 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:41.138135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:41.138076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:41.838732 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:41.838682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:41.941487 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:41.941454 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:42.041589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.041554 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:42.375279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.375247 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:48:42.375741 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.375458 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" podUID="366c2341-ea18-459e-a3c5-da9aaeb53155" containerName="maas-api" containerID="cri-o://89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91" gracePeriod=30 Nov 03 20:48:42.520949 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.520918 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:42.622140 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.622082 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:42.636477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.636413 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:48:42.745967 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.745930 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\") pod \"366c2341-ea18-459e-a3c5-da9aaeb53155\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " Nov 03 20:48:42.746234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.745983 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\" (UniqueName: \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\") pod \"366c2341-ea18-459e-a3c5-da9aaeb53155\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " Nov 03 20:48:42.746234 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.746094 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\") pod \"366c2341-ea18-459e-a3c5-da9aaeb53155\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " Nov 03 20:48:42.746384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.746358 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca" pod "366c2341-ea18-459e-a3c5-da9aaeb53155" (UID: "366c2341-ea18-459e-a3c5-da9aaeb53155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:42.746611 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.746588 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\") pod \"366c2341-ea18-459e-a3c5-da9aaeb53155\" (UID: \"366c2341-ea18-459e-a3c5-da9aaeb53155\") " Nov 03 20:48:42.746936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.746915 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/366c2341-ea18-459e-a3c5-da9aaeb53155-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:42.748295 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.748268 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7" pod "366c2341-ea18-459e-a3c5-da9aaeb53155" (UID: "366c2341-ea18-459e-a3c5-da9aaeb53155"). InnerVolumeSpecName "kube-api-access-58hf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:42.748384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.748294 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs" pod "366c2341-ea18-459e-a3c5-da9aaeb53155" (UID: "366c2341-ea18-459e-a3c5-da9aaeb53155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:42.748717 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.748683 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls" pod "366c2341-ea18-459e-a3c5-da9aaeb53155" (UID: "366c2341-ea18-459e-a3c5-da9aaeb53155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:42.768754 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.768728 5954 generic.go:356] "Generic (PLEG): container finished" podID="366c2341-ea18-459e-a3c5-da9aaeb53155" containerID="89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91" exitCode=0 Nov 03 20:48:42.768866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.768783 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" Nov 03 20:48:42.768866 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.768807 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" event={"ID":"366c2341-ea18-459e-a3c5-da9aaeb53155","Type":"ContainerDied","Data":"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91"} Nov 03 20:48:42.768955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.768882 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz" event={"ID":"366c2341-ea18-459e-a3c5-da9aaeb53155","Type":"ContainerDied","Data":"14f8e65b4a652dc976160dd6764e40cd2bd63d9b5cb638f6936f6cc253016b17"} Nov 03 20:48:42.768955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.768905 5954 scope.go:122] "RemoveContainer" containerID="89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91" Nov 03 20:48:42.780770 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.780752 5954 scope.go:122] "RemoveContainer" containerID="89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91" Nov 03 20:48:42.781059 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:48:42.781040 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91\": container with ID starting with 89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91 not found: ID does not exist" containerID="89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91" Nov 03 20:48:42.781179 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.781070 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91"} err="failed to get container status \"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91\": rpc error: code = NotFound desc = could not find container \"89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91\": container with ID starting with 89a5cc00ae2158f6a981e5cdd543d8cb643edfa2929f74e5192bdc4172380c91 not found: ID does not exist" Nov 03 20:48:42.793991 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.793968 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:48:42.798985 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.798964 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e95cf7-54dbf79cc5-5bwrz"] Nov 03 20:48:42.848166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.848140 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hf7\" (UniqueName: \"kubernetes.io/projected/366c2341-ea18-459e-a3c5-da9aaeb53155-kube-api-access-58hf7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:42.848166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.848166 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:42.848339 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:42.848176 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/366c2341-ea18-459e-a3c5-da9aaeb53155-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:43.161620 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:43.161586 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:43.403765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:43.403731 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366c2341-ea18-459e-a3c5-da9aaeb53155" path="/var/lib/kubelet/pods/366c2341-ea18-459e-a3c5-da9aaeb53155/volumes" Nov 03 20:48:44.132485 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:44.132449 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:48:48.021250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:48.021217 5954 scope.go:122] "RemoveContainer" containerID="a82518b04a6411be88d7fa8b1478efa91b24956135b386fb607918752bc897cc" Nov 03 20:48:49.001580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:49.001549 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.100309 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:49.100259 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.373861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:49.373823 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.658602 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:49.658513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:50.627161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.627100 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5"] Nov 03 20:48:50.631839 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.631814 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qr7m5"] Nov 03 20:48:50.651646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.651621 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw"] Nov 03 20:48:50.652062 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.652048 5954 cpu_manager.go:414] "containerMap: removing container" podUID="366c2341-ea18-459e-a3c5-da9aaeb53155" containerName="maas-api" Nov 03 20:48:50.652136 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.652063 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366c2341-ea18-459e-a3c5-da9aaeb53155" containerName="maas-api" Nov 03 20:48:50.652181 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.652173 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="366c2341-ea18-459e-a3c5-da9aaeb53155" containerName="maas-api" Nov 03 20:48:50.656740 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.656723 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.666351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.666273 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:48:50.666351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.666302 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:48:50.666351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.666303 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:48:50.666351 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.666273 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:48:50.666608 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.666526 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw"] Nov 03 20:48:50.732161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.732098 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.732337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.732292 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.732389 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.732334 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbm2d\" (UniqueName: \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.833516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.833476 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbm2d\" (UniqueName: \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.833700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.833542 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.833700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.833631 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.834183 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.834160 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.835880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.835858 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.846271 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.846252 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbm2d\" (UniqueName: \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\") pod \"maas-api-revoke-sub-3d4156f1-kfbsw\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:50.971076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:50.970999 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:51.245781 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:51.245755 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw"] Nov 03 20:48:51.248329 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:48:51.248300 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a603b1_4a9a_4619_9a87_d9730d7c990d.slice/crio-ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5 WatchSource:0}: Error finding container ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5: Status 404 returned error can't find the container with id ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5 Nov 03 20:48:51.404307 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:51.404275 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9dbac80-b4bd-45d2-b411-b56f3adfc55e" path="/var/lib/kubelet/pods/c9dbac80-b4bd-45d2-b411-b56f3adfc55e/volumes" Nov 03 20:48:51.816757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:51.816663 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" event={"ID":"e2a603b1-4a9a-4619-9a87-d9730d7c990d","Type":"ContainerStarted","Data":"066c3a5f5de4e2fa5b1415ab1b37cb0789fe8a1d368d1f4d9fdde421091f16ca"} Nov 03 20:48:51.816757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:51.816729 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" event={"ID":"e2a603b1-4a9a-4619-9a87-d9730d7c990d","Type":"ContainerStarted","Data":"ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5"} Nov 03 20:48:51.836529 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:51.836482 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" podStartSLOduration=1.8364659190000001 podStartE2EDuration="1.836465919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:51.835425352 +0000 UTC m=+3072.560432951" watchObservedRunningTime="2026-11-03 20:48:51.836465919 +0000 UTC m=+3072.561473514" Nov 03 20:48:52.823823 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:52.823791 5954 generic.go:356] "Generic (PLEG): container finished" podID="e2a603b1-4a9a-4619-9a87-d9730d7c990d" containerID="066c3a5f5de4e2fa5b1415ab1b37cb0789fe8a1d368d1f4d9fdde421091f16ca" exitCode=0 Nov 03 20:48:52.824237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:52.823877 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" event={"ID":"e2a603b1-4a9a-4619-9a87-d9730d7c990d","Type":"ContainerDied","Data":"066c3a5f5de4e2fa5b1415ab1b37cb0789fe8a1d368d1f4d9fdde421091f16ca"} Nov 03 20:48:53.963192 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:53.963166 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:54.063006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.062975 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\") pod \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " Nov 03 20:48:54.063226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.063087 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\") pod \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " Nov 03 20:48:54.063226 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.063206 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\" (UniqueName: \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\") pod \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\" (UID: \"e2a603b1-4a9a-4619-9a87-d9730d7c990d\") " Nov 03 20:48:54.063469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.063442 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca" pod "e2a603b1-4a9a-4619-9a87-d9730d7c990d" (UID: "e2a603b1-4a9a-4619-9a87-d9730d7c990d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:54.065236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.065209 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d" pod "e2a603b1-4a9a-4619-9a87-d9730d7c990d" (UID: "e2a603b1-4a9a-4619-9a87-d9730d7c990d"). InnerVolumeSpecName "kube-api-access-hbm2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:54.065353 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.065252 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca" pod "e2a603b1-4a9a-4619-9a87-d9730d7c990d" (UID: "e2a603b1-4a9a-4619-9a87-d9730d7c990d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:54.164632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.164597 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbm2d\" (UniqueName: \"kubernetes.io/projected/e2a603b1-4a9a-4619-9a87-d9730d7c990d-kube-api-access-hbm2d\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.164632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.164625 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.164632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.164637 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a603b1-4a9a-4619-9a87-d9730d7c990d-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.835000 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.834967 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" Nov 03 20:48:54.835197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.834971 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw" event={"ID":"e2a603b1-4a9a-4619-9a87-d9730d7c990d","Type":"ContainerDied","Data":"ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5"} Nov 03 20:48:54.835197 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:48:54.835141 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac9b5a5730c95c5762ba75c2e64a3264d956e83b9aba50056dead9b668822bf5" Nov 03 20:49:00.257710 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:49:00.257676 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:49:02.433178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:02.433142 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:49:07.528435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:07.528393 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:49:12.927024 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:12.926984 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:14.451023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:14.450983 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:20.385972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:20.385933 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:30.256926 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:49:30.256883 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:49:47.897984 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:47.897936 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" podUID="d161bc542803fe55ee71704a451b4d4d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:49:49.721050 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:49:49.721000 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:00.257461 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:50:00.257420 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:50:30.257303 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:50:30.257217 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:50:31.527973 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:31.527940 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:50:44.400759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.400728 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4"] Nov 03 20:50:44.401230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.401193 5954 cpu_manager.go:414] "containerMap: removing container" podUID="e2a603b1-4a9a-4619-9a87-d9730d7c990d" containerName="revoke-keys" Nov 03 20:50:44.401230 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.401207 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a603b1-4a9a-4619-9a87-d9730d7c990d" containerName="revoke-keys" Nov 03 20:50:44.401320 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.401288 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a603b1-4a9a-4619-9a87-d9730d7c990d" containerName="revoke-keys" Nov 03 20:50:44.404519 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.404502 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.409542 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.409507 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:44.410635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.410613 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:50:44.410861 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.410834 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:50:44.410980 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.410965 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:50:44.413378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.413015 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4"] Nov 03 20:50:44.509511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.509476 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.509703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.509518 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4kqd\" (UniqueName: \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.509789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.509710 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.610581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.610533 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.610792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.610622 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.610792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.610650 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4kqd\" (UniqueName: \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.611338 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.611314 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.612923 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.612899 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.622748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.622724 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4kqd\" (UniqueName: \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\") pod \"maas-api-revoke-sub-d576e68b-2gzq4\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.721012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.720924 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:44.994061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:44.994035 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4"] Nov 03 20:50:44.996309 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:50:44.996281 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91de4c5b_f078_410f_b55a_3705fe52c944.slice/crio-f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df WatchSource:0}: Error finding container f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df: Status 404 returned error can't find the container with id f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df Nov 03 20:50:45.395525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:45.395485 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" event={"ID":"91de4c5b-f078-410f-b55a-3705fe52c944","Type":"ContainerStarted","Data":"0bca03cc46cd0e97d2953132b0851b0d676ab7d72a3d6e4930e4d391aadfa675"} Nov 03 20:50:45.395688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:45.395547 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" event={"ID":"91de4c5b-f078-410f-b55a-3705fe52c944","Type":"ContainerStarted","Data":"f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df"} Nov 03 20:50:45.413416 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:45.413367 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" podStartSLOduration=1.413352995 podStartE2EDuration="1.413352995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:45.412393918 +0000 UTC m=+3186.137401510" watchObservedRunningTime="2026-11-03 20:50:45.413352995 +0000 UTC m=+3186.138360582" Nov 03 20:50:46.402672 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:46.402642 5954 generic.go:356] "Generic (PLEG): container finished" podID="91de4c5b-f078-410f-b55a-3705fe52c944" containerID="0bca03cc46cd0e97d2953132b0851b0d676ab7d72a3d6e4930e4d391aadfa675" exitCode=0 Nov 03 20:50:46.402851 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:46.402727 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" event={"ID":"91de4c5b-f078-410f-b55a-3705fe52c944","Type":"ContainerDied","Data":"0bca03cc46cd0e97d2953132b0851b0d676ab7d72a3d6e4930e4d391aadfa675"} Nov 03 20:50:47.537237 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.537213 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:47.640180 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.640128 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\") pod \"91de4c5b-f078-410f-b55a-3705fe52c944\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " Nov 03 20:50:47.640395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.640252 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\") pod \"91de4c5b-f078-410f-b55a-3705fe52c944\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " Nov 03 20:50:47.640395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.640367 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\" (UniqueName: \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\") pod \"91de4c5b-f078-410f-b55a-3705fe52c944\" (UID: \"91de4c5b-f078-410f-b55a-3705fe52c944\") " Nov 03 20:50:47.640589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.640564 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca" pod "91de4c5b-f078-410f-b55a-3705fe52c944" (UID: "91de4c5b-f078-410f-b55a-3705fe52c944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:47.640742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.640720 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:47.642405 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.642381 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca" pod "91de4c5b-f078-410f-b55a-3705fe52c944" (UID: "91de4c5b-f078-410f-b55a-3705fe52c944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:47.642502 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.642476 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd" pod "91de4c5b-f078-410f-b55a-3705fe52c944" (UID: "91de4c5b-f078-410f-b55a-3705fe52c944"). InnerVolumeSpecName "kube-api-access-k4kqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:47.741810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.741722 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91de4c5b-f078-410f-b55a-3705fe52c944-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:47.741810 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:47.741760 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4kqd\" (UniqueName: \"kubernetes.io/projected/91de4c5b-f078-410f-b55a-3705fe52c944-kube-api-access-k4kqd\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:48.413565 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:48.413532 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" Nov 03 20:50:48.413761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:48.413582 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4" event={"ID":"91de4c5b-f078-410f-b55a-3705fe52c944","Type":"ContainerDied","Data":"f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df"} Nov 03 20:50:48.413761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:48.413627 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f28c8a1f9c92527d192dec2860b113f060e3400423acf5b4b870b1245fbf68df" Nov 03 20:50:51.346745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:51.346709 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:55.353537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:55.353494 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:50:59.505909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:50:59.505875 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:51:00.258481 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:51:00.258437 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:51:03.478753 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.478723 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl"] Nov 03 20:51:03.479231 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.479215 5954 cpu_manager.go:414] "containerMap: removing container" podUID="91de4c5b-f078-410f-b55a-3705fe52c944" containerName="revoke-keys" Nov 03 20:51:03.479293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.479232 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91de4c5b-f078-410f-b55a-3705fe52c944" containerName="revoke-keys" Nov 03 20:51:03.479334 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.479299 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="91de4c5b-f078-410f-b55a-3705fe52c944" containerName="revoke-keys" Nov 03 20:51:03.482429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.482410 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.490281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.490257 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:51:03.490913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.490891 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:03.492577 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.492551 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl"] Nov 03 20:51:03.513489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.513452 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:51:03.513489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.513460 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:51:03.523865 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.523838 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p486v\" (UniqueName: \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.524044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.524019 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.524097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.524080 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.625403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.625370 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p486v\" (UniqueName: \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.625593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.625459 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.625593 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.625488 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.626170 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.626148 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.627736 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.627718 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.637479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.637454 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p486v\" (UniqueName: \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\") pod \"maas-api-revoke-sub-d2bc93bc-2pzpl\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:03.798135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:03.798034 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:04.072579 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:04.072551 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl"] Nov 03 20:51:04.074766 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:51:04.074734 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4999a164_7215_4eb5_9c6e_b5a53bc76b69.slice/crio-d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca WatchSource:0}: Error finding container d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca: Status 404 returned error can't find the container with id d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca Nov 03 20:51:04.519968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:04.519920 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" event={"ID":"4999a164-7215-4eb5-9c6e-b5a53bc76b69","Type":"ContainerStarted","Data":"e94f564f5520c8099e0ac3c691a97ee944240520ef8098392dda183a30ee04fb"} Nov 03 20:51:04.520470 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:04.520016 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" event={"ID":"4999a164-7215-4eb5-9c6e-b5a53bc76b69","Type":"ContainerStarted","Data":"d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca"} Nov 03 20:51:04.539586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:04.539542 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" podStartSLOduration=1.539529984 podStartE2EDuration="1.539529984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:04.538007506 +0000 UTC m=+3205.263015101" watchObservedRunningTime="2026-11-03 20:51:04.539529984 +0000 UTC m=+3205.264537581" Nov 03 20:51:05.530282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:05.530245 5954 generic.go:356] "Generic (PLEG): container finished" podID="4999a164-7215-4eb5-9c6e-b5a53bc76b69" containerID="e94f564f5520c8099e0ac3c691a97ee944240520ef8098392dda183a30ee04fb" exitCode=0 Nov 03 20:51:05.530282 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:05.530285 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" event={"ID":"4999a164-7215-4eb5-9c6e-b5a53bc76b69","Type":"ContainerDied","Data":"e94f564f5520c8099e0ac3c691a97ee944240520ef8098392dda183a30ee04fb"} Nov 03 20:51:06.675983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.675958 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:06.756469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.756427 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\" (UniqueName: \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\") pod \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " Nov 03 20:51:06.756674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.756494 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\") pod \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " Nov 03 20:51:06.756674 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.756520 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\") pod \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\" (UID: \"4999a164-7215-4eb5-9c6e-b5a53bc76b69\") " Nov 03 20:51:06.756925 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.756896 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca" pod "4999a164-7215-4eb5-9c6e-b5a53bc76b69" (UID: "4999a164-7215-4eb5-9c6e-b5a53bc76b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:06.758624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.758598 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca" pod "4999a164-7215-4eb5-9c6e-b5a53bc76b69" (UID: "4999a164-7215-4eb5-9c6e-b5a53bc76b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:06.758624 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.758616 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v" pod "4999a164-7215-4eb5-9c6e-b5a53bc76b69" (UID: "4999a164-7215-4eb5-9c6e-b5a53bc76b69"). InnerVolumeSpecName "kube-api-access-p486v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:06.857478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.857440 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:06.857478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.857473 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p486v\" (UniqueName: \"kubernetes.io/projected/4999a164-7215-4eb5-9c6e-b5a53bc76b69-kube-api-access-p486v\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:06.857478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:06.857483 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4999a164-7215-4eb5-9c6e-b5a53bc76b69-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.541303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:07.541262 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" event={"ID":"4999a164-7215-4eb5-9c6e-b5a53bc76b69","Type":"ContainerDied","Data":"d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca"} Nov 03 20:51:07.541303 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:07.541287 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl" Nov 03 20:51:07.541535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:07.541325 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1eb716eb62b68715a4d4a67fe6b3e3f9a7f9b61e6157da60b030a2904d22bca" Nov 03 20:51:13.794604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:13.794568 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:13.895297 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:13.895261 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:14.436689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:14.436656 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:51:21.434560 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:21.434529 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.218036 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:27.217998 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.683522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:27.683485 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.795544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:27.795513 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:30.256816 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:51:30.256778 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:51:43.345600 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:43.345563 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.554650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:43.554616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.491095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.491060 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:51:44.491661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.491538 5954 cpu_manager.go:414] "containerMap: removing container" podUID="4999a164-7215-4eb5-9c6e-b5a53bc76b69" containerName="revoke-keys" Nov 03 20:51:44.491661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.491551 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4999a164-7215-4eb5-9c6e-b5a53bc76b69" containerName="revoke-keys" Nov 03 20:51:44.491661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.491647 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="4999a164-7215-4eb5-9c6e-b5a53bc76b69" containerName="revoke-keys" Nov 03 20:51:44.495347 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.495316 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.509120 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.509077 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:51:44.509313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.509248 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:51:44.510034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.510011 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:51:44.518147 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.518099 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.518404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.518383 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.518598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.518574 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.518734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.518694 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkw54\" (UniqueName: \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.552447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.552414 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.554315 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.554291 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-466ab7-serving-cert\"" Nov 03 20:51:44.554558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.554532 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:51:44.554714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.554674 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:51:44.620305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.620272 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkw54\" (UniqueName: \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.620484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.620376 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.620484 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.620435 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.620563 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.620482 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.621079 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.621043 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.622948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.622920 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.623065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.622956 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.638264 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.638234 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkw54\" (UniqueName: \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\") pod \"maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:44.672617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.672574 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.812802 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:44.812717 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:45.092137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.092094 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:51:45.094985 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:51:45.094948 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139bf28c_de4c_4577_90d7_c0c047f7f711.slice/crio-3ac858cb0c55824b3a2a5f30748d66c74257c0d7fd99b2c06096b781a99013c8 WatchSource:0}: Error finding container 3ac858cb0c55824b3a2a5f30748d66c74257c0d7fd99b2c06096b781a99013c8: Status 404 returned error can't find the container with id 3ac858cb0c55824b3a2a5f30748d66c74257c0d7fd99b2c06096b781a99013c8 Nov 03 20:51:45.357073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.357043 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.745629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.745594 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" event={"ID":"139bf28c-de4c-4577-90d7-c0c047f7f711","Type":"ContainerStarted","Data":"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c"} Nov 03 20:51:45.746038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.745650 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" event={"ID":"139bf28c-de4c-4577-90d7-c0c047f7f711","Type":"ContainerStarted","Data":"3ac858cb0c55824b3a2a5f30748d66c74257c0d7fd99b2c06096b781a99013c8"} Nov 03 20:51:45.746038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.745693 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:45.765913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:45.765857 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" podStartSLOduration=1.267104453 podStartE2EDuration="1.765841703s" totalImagesPullingTime="498.73725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.764141246 +0000 UTC m=+3246.489148839" watchObservedRunningTime="2026-11-03 20:51:45.765841703 +0000 UTC m=+3246.490849299" Nov 03 20:51:46.058936 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:46.058857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:46.355686 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:46.355649 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:51:46.793943 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:46.793817 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:47.640772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:47.640691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:48.251207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:48.251172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:49.146690 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:49.146660 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:49.869447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:49.869410 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:49.968888 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:49.968816 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:50.559229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:50.559193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:51.265838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:51.265807 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:51.755746 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:51.755716 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:51:51.954087 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:51.954050 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:52.729763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:52.729702 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.243001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:53.242953 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.462663 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:53.462627 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.859043 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:53.858994 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.961982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:53.961941 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:54.066016 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:54.065975 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:54.231550 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:54.231460 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:55.052632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:55.052595 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:55.763195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:55.763137 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:55.869207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:55.869157 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:56.562929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:56.562901 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:57.244898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:57.244866 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:57.470275 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:57.470244 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:51:57.858627 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:57.858588 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:58.036625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:58.036592 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:58.856513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:58.856479 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.656217 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:51:59.656146 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:00.257479 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:52:00.257444 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:52:00.456287 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:00.456257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:01.258051 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:01.258000 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:01.955776 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:01.955730 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:02.748506 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:02.748472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:03.443825 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:03.443781 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:04.237184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:04.237150 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:04.941848 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:04.941820 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:05.654507 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:05.654472 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:06.451489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:06.451461 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:07.144340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:07.144307 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:07.854850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:07.854814 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:08.568508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:08.568466 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:09.246722 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:09.246685 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:09.962346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:09.962306 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:10.649477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:10.649448 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:11.434161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:11.434125 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:12.153824 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:12.153788 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:12.563742 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:12.563661 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:52:12.961814 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:12.961782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:13.804795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:13.804756 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.539974 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:14.539937 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.250795 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:15.250748 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:16.038777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:16.038720 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:16.736738 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:16.736696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:17.441355 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:17.441321 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.141154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:18.141089 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.892940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:18.892903 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.540786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:19.540751 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.236203 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:20.236163 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.935860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:20.935829 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:21.635229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:21.635193 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.353510 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:22.353474 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.154808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:23.154777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.848276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:23.848233 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:24.557481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:24.557446 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:25.439006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:25.438974 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.143023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:26.142966 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.620184 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:26.620140 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.720387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:26.720353 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.936891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:26.936808 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:27.645054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:27.645013 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:28.344361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:28.344298 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:29.051390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:29.051350 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:29.744466 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:29.744431 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:30.260467 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:52:30.260423 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:52:30.435545 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:30.435510 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:31.142101 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:31.142068 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:31.848465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:31.848430 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:32.549677 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:32.549645 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:33.251972 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:33.251938 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:33.946940 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:33.946870 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:34.645963 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:34.645926 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:35.352797 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:35.352762 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:36.185998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:36.185954 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:36.286168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:36.286053 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:36.422747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:36.422710 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:52:37.096556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:37.096523 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:37.864636 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:37.864602 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:38.544508 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:38.544474 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.255894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:39.255854 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.952965 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:39.952932 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:40.660886 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:40.660852 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:41.195308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:41.195272 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:41.360127 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:41.360076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:42.051478 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:42.051445 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:42.740590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:42.740557 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:43.442648 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:43.442616 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:44.147646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:44.147612 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:44.844775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:44.844739 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:45.546673 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:45.546638 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.256298 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:46.256265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.564317 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:46.564245 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.664072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:46.664039 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.951069 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:46.951039 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.478589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.478543 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:52:47.479097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.478799 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" podUID="139bf28c-de4c-4577-90d7-c0c047f7f711" containerName="maas-api" containerID="cri-o://ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c" gracePeriod=30 Nov 03 20:52:47.546714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.546682 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.748885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.748789 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.855544 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.855508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.945013 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:47.944986 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:52:48.023629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.023549 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\") pod \"139bf28c-de4c-4577-90d7-c0c047f7f711\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " Nov 03 20:52:48.023799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.023671 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\" (UniqueName: \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\") pod \"139bf28c-de4c-4577-90d7-c0c047f7f711\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " Nov 03 20:52:48.023799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.023727 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\") pod \"139bf28c-de4c-4577-90d7-c0c047f7f711\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " Nov 03 20:52:48.023920 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.023797 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\") pod \"139bf28c-de4c-4577-90d7-c0c047f7f711\" (UID: \"139bf28c-de4c-4577-90d7-c0c047f7f711\") " Nov 03 20:52:48.024245 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.024216 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca" pod "139bf28c-de4c-4577-90d7-c0c047f7f711" (UID: "139bf28c-de4c-4577-90d7-c0c047f7f711"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:48.025898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.025857 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls" pod "139bf28c-de4c-4577-90d7-c0c047f7f711" (UID: "139bf28c-de4c-4577-90d7-c0c047f7f711"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:48.026058 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.026037 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs" pod "139bf28c-de4c-4577-90d7-c0c047f7f711" (UID: "139bf28c-de4c-4577-90d7-c0c047f7f711"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:48.026159 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.026085 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54" pod "139bf28c-de4c-4577-90d7-c0c047f7f711" (UID: "139bf28c-de4c-4577-90d7-c0c047f7f711"). InnerVolumeSpecName "kube-api-access-lkw54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:48.076406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.076368 5954 generic.go:356] "Generic (PLEG): container finished" podID="139bf28c-de4c-4577-90d7-c0c047f7f711" containerID="ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c" exitCode=0 Nov 03 20:52:48.076562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.076413 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" Nov 03 20:52:48.076562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.076452 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" event={"ID":"139bf28c-de4c-4577-90d7-c0c047f7f711","Type":"ContainerDied","Data":"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c"} Nov 03 20:52:48.076562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.076507 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2" event={"ID":"139bf28c-de4c-4577-90d7-c0c047f7f711","Type":"ContainerDied","Data":"3ac858cb0c55824b3a2a5f30748d66c74257c0d7fd99b2c06096b781a99013c8"} Nov 03 20:52:48.076562 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.076533 5954 scope.go:122] "RemoveContainer" containerID="ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c" Nov 03 20:52:48.086007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.085986 5954 scope.go:122] "RemoveContainer" containerID="ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c" Nov 03 20:52:48.086318 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:52:48.086298 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c\": container with ID starting with ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c not found: ID does not exist" containerID="ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c" Nov 03 20:52:48.086388 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.086329 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c"} err="failed to get container status \"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c\": rpc error: code = NotFound desc = could not find container \"ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c\": container with ID starting with ba47e8d74f86f9d0cdef8199a352b2fc6be3ecab658540852a7d25439ea3e30c not found: ID does not exist" Nov 03 20:52:48.100381 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.100345 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:52:48.102864 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.102837 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-466ab7-5b676b77cd-xjvp2"] Nov 03 20:52:48.125703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.125661 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkw54\" (UniqueName: \"kubernetes.io/projected/139bf28c-de4c-4577-90d7-c0c047f7f711-kube-api-access-lkw54\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:48.125703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.125701 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:48.125911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.125717 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/139bf28c-de4c-4577-90d7-c0c047f7f711-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:48.125911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.125734 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/139bf28c-de4c-4577-90d7-c0c047f7f711-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:48.325761 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:48.325683 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:49.403248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:49.403214 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139bf28c-de4c-4577-90d7-c0c047f7f711" path="/var/lib/kubelet/pods/139bf28c-de4c-4577-90d7-c0c047f7f711/volumes" Nov 03 20:52:49.453975 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:49.453934 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:52:49.920910 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:49.920875 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:54.261233 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:54.261184 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:54.357318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:54.357286 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:54.635061 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:54.635028 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:58.306927 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:58.306894 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:59.950134 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:52:59.950081 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:00.256601 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:53:00.256521 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:53:15.458335 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:15.458302 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:16.976554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:16.976519 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:19.069745 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:19.069699 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.455131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:19.454997 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.791302 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:19.791216 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:19.791582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:19.791554 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:25.354002 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:25.353965 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:30.257645 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:53:30.257609 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:53:32.360845 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.360810 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8"] Nov 03 20:53:32.361318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.361292 5954 cpu_manager.go:414] "containerMap: removing container" podUID="139bf28c-de4c-4577-90d7-c0c047f7f711" containerName="maas-api" Nov 03 20:53:32.361318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.361309 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139bf28c-de4c-4577-90d7-c0c047f7f711" containerName="maas-api" Nov 03 20:53:32.361451 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.361411 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="139bf28c-de4c-4577-90d7-c0c047f7f711" containerName="maas-api" Nov 03 20:53:32.365070 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.365052 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.370606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.370584 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:53:32.370777 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.370590 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:32.370901 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.370657 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:53:32.372015 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.371983 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:53:32.374597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.374577 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8"] Nov 03 20:53:32.454929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.454892 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.455150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.454964 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.455150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.455070 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwxn\" (UniqueName: \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.556066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.556032 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.556066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.556075 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwxn\" (UniqueName: \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.556321 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.556195 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.556813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.556789 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.558475 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.558455 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.568604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.568583 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwxn\" (UniqueName: \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\") pod \"maas-api-revoke-sub-452b6d7b-m6dx8\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.681629 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.681540 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:32.941336 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.941265 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:32.960063 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:32.960036 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8"] Nov 03 20:53:32.962456 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:53:32.962428 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d590b1_2bab_4faf_99f8_a1de6503eca3.slice/crio-c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7 WatchSource:0}: Error finding container c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7: Status 404 returned error can't find the container with id c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7 Nov 03 20:53:33.360370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:33.360329 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" event={"ID":"84d590b1-2bab-4faf-99f8-a1de6503eca3","Type":"ContainerStarted","Data":"a1ed98ee69cc06b52f74b71342a9c0a886be442816b0adce98bb3fc5e9841506"} Nov 03 20:53:33.360558 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:33.360390 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" event={"ID":"84d590b1-2bab-4faf-99f8-a1de6503eca3","Type":"ContainerStarted","Data":"c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7"} Nov 03 20:53:33.380513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:33.380459 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" podStartSLOduration=1.380443101 podStartE2EDuration="1.380443101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:33.377896378 +0000 UTC m=+3354.102903976" watchObservedRunningTime="2026-11-03 20:53:33.380443101 +0000 UTC m=+3354.105450696" Nov 03 20:53:34.369224 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:34.369187 5954 generic.go:356] "Generic (PLEG): container finished" podID="84d590b1-2bab-4faf-99f8-a1de6503eca3" containerID="a1ed98ee69cc06b52f74b71342a9c0a886be442816b0adce98bb3fc5e9841506" exitCode=0 Nov 03 20:53:34.369490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:34.369293 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" event={"ID":"84d590b1-2bab-4faf-99f8-a1de6503eca3","Type":"ContainerDied","Data":"a1ed98ee69cc06b52f74b71342a9c0a886be442816b0adce98bb3fc5e9841506"} Nov 03 20:53:35.755001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.754690 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:35.897007 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.896972 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\") pod \"84d590b1-2bab-4faf-99f8-a1de6503eca3\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " Nov 03 20:53:35.897250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.897030 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\" (UniqueName: \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\") pod \"84d590b1-2bab-4faf-99f8-a1de6503eca3\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " Nov 03 20:53:35.897250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.897220 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\") pod \"84d590b1-2bab-4faf-99f8-a1de6503eca3\" (UID: \"84d590b1-2bab-4faf-99f8-a1de6503eca3\") " Nov 03 20:53:35.897490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.897462 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca" pod "84d590b1-2bab-4faf-99f8-a1de6503eca3" (UID: "84d590b1-2bab-4faf-99f8-a1de6503eca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:35.897584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.897567 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:35.899326 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.899298 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn" pod "84d590b1-2bab-4faf-99f8-a1de6503eca3" (UID: "84d590b1-2bab-4faf-99f8-a1de6503eca3"). InnerVolumeSpecName "kube-api-access-sxwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:35.899450 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.899375 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca" pod "84d590b1-2bab-4faf-99f8-a1de6503eca3" (UID: "84d590b1-2bab-4faf-99f8-a1de6503eca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:35.998785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.998694 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxwxn\" (UniqueName: \"kubernetes.io/projected/84d590b1-2bab-4faf-99f8-a1de6503eca3-kube-api-access-sxwxn\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:35.998785 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:35.998729 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d590b1-2bab-4faf-99f8-a1de6503eca3-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:36.395193 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:36.395149 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" event={"ID":"84d590b1-2bab-4faf-99f8-a1de6503eca3","Type":"ContainerDied","Data":"c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7"} Nov 03 20:53:36.395378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:36.395216 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29adcf47311a0086cb771600286f79f69c2921c31fa176e58691102c26587d7" Nov 03 20:53:36.395378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:36.395289 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8" Nov 03 20:53:43.248119 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:43.248068 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.848747 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:43.848709 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:44.619078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:44.619043 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:45.348592 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.348558 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4"] Nov 03 20:53:45.349034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.349021 5954 cpu_manager.go:414] "containerMap: removing container" podUID="84d590b1-2bab-4faf-99f8-a1de6503eca3" containerName="revoke-keys" Nov 03 20:53:45.349097 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.349036 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d590b1-2bab-4faf-99f8-a1de6503eca3" containerName="revoke-keys" Nov 03 20:53:45.349152 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.349136 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d590b1-2bab-4faf-99f8-a1de6503eca3" containerName="revoke-keys" Nov 03 20:53:45.352340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.352319 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.361054 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.361028 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4"] Nov 03 20:53:45.361843 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.361817 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:53:45.361962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.361897 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:45.361962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.361919 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:53:45.362102 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.362076 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:53:45.501734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.501698 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsz7\" (UniqueName: \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.501954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.501746 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.501954 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.501875 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.603324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.603285 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.603540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.603427 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsz7\" (UniqueName: \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.603540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.603464 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.604076 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.604054 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.605775 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.605754 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.615909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.615884 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsz7\" (UniqueName: \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\") pod \"maas-api-revoke-sub-3e31a506-g8xd4\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.667998 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.667961 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:45.943652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:45.943523 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4"] Nov 03 20:53:45.946118 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:53:45.946077 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bb1232_ea47_4c42_b06c_7b2b62f4719c.slice/crio-b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b WatchSource:0}: Error finding container b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b: Status 404 returned error can't find the container with id b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b Nov 03 20:53:46.444504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:46.444464 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" event={"ID":"a2bb1232-ea47-4c42-b06c-7b2b62f4719c","Type":"ContainerStarted","Data":"278525ab3fed5f06d1a73ca82b44290425b9df49ce5e609eaceef1bdf32da384"} Nov 03 20:53:46.444680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:46.444527 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" event={"ID":"a2bb1232-ea47-4c42-b06c-7b2b62f4719c","Type":"ContainerStarted","Data":"b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b"} Nov 03 20:53:46.517318 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:46.517234 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" podStartSLOduration=1.5172189170000001 podStartE2EDuration="1.517218917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:46.515309001 +0000 UTC m=+3367.240316596" watchObservedRunningTime="2026-11-03 20:53:46.517218917 +0000 UTC m=+3367.242226702" Nov 03 20:53:47.453996 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:47.453953 5954 generic.go:356] "Generic (PLEG): container finished" podID="a2bb1232-ea47-4c42-b06c-7b2b62f4719c" containerID="278525ab3fed5f06d1a73ca82b44290425b9df49ce5e609eaceef1bdf32da384" exitCode=0 Nov 03 20:53:47.454580 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:47.454015 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" event={"ID":"a2bb1232-ea47-4c42-b06c-7b2b62f4719c","Type":"ContainerDied","Data":"278525ab3fed5f06d1a73ca82b44290425b9df49ce5e609eaceef1bdf32da384"} Nov 03 20:53:48.079912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.079874 5954 scope.go:122] "RemoveContainer" containerID="4ca75c20a15977cac03282ccdd02c530b9e11fdd43c237b5a3a138c7461912dc" Nov 03 20:53:48.816787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.816474 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:48.991482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.991431 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\") pod \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " Nov 03 20:53:48.991708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.991507 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\") pod \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " Nov 03 20:53:48.991708 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.991537 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\" (UniqueName: \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\") pod \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\" (UID: \"a2bb1232-ea47-4c42-b06c-7b2b62f4719c\") " Nov 03 20:53:48.992403 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.992377 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca" pod "a2bb1232-ea47-4c42-b06c-7b2b62f4719c" (UID: "a2bb1232-ea47-4c42-b06c-7b2b62f4719c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:48.993815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.993791 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca" pod "a2bb1232-ea47-4c42-b06c-7b2b62f4719c" (UID: "a2bb1232-ea47-4c42-b06c-7b2b62f4719c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:48.993885 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:48.993838 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7" pod "a2bb1232-ea47-4c42-b06c-7b2b62f4719c" (UID: "a2bb1232-ea47-4c42-b06c-7b2b62f4719c"). InnerVolumeSpecName "kube-api-access-4tsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:49.092880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.092787 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:49.092880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.092831 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:49.092880 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.092850 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tsz7\" (UniqueName: \"kubernetes.io/projected/a2bb1232-ea47-4c42-b06c-7b2b62f4719c-kube-api-access-4tsz7\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:49.445131 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.431372 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:49.488075 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.488023 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" event={"ID":"a2bb1232-ea47-4c42-b06c-7b2b62f4719c","Type":"ContainerDied","Data":"b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b"} Nov 03 20:53:49.488377 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.488356 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0c24523df06e48c9f1d36eefb72926bae9c4b67c7cd6ee59ab6da87862a011b" Nov 03 20:53:49.488590 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.488574 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4" Nov 03 20:53:49.793313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.793222 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:53:49.793313 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.793278 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.977899 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:49.977859 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:54.023862 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:54.023822 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw"] Nov 03 20:53:54.026811 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:54.026787 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kfbsw"] Nov 03 20:53:55.403983 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:55.403951 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a603b1-4a9a-4619-9a87-d9730d7c990d" path="/var/lib/kubelet/pods/e2a603b1-4a9a-4619-9a87-d9730d7c990d/volumes" Nov 03 20:53:56.168194 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:56.168161 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:53:57.186044 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:57.186010 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:57.277909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:53:57.277873 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.303520 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:54:00.303464 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:54:00.426085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:00.426039 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:54:06.760023 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.759985 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg"] Nov 03 20:54:06.760488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.760464 5954 cpu_manager.go:414] "containerMap: removing container" podUID="a2bb1232-ea47-4c42-b06c-7b2b62f4719c" containerName="revoke-keys" Nov 03 20:54:06.760488 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.760478 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bb1232-ea47-4c42-b06c-7b2b62f4719c" containerName="revoke-keys" Nov 03 20:54:06.760617 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.760549 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bb1232-ea47-4c42-b06c-7b2b62f4719c" containerName="revoke-keys" Nov 03 20:54:06.765096 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.765078 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.771919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.771491 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:06.771919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.771557 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:54:06.771919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.771638 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:54:06.771919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.771713 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:54:06.772268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.772244 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg"] Nov 03 20:54:06.892504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.892465 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tzt\" (UniqueName: \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.892710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.892513 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.892710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.892618 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.993417 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.993379 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.993586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.993512 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tzt\" (UniqueName: \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.993586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.993564 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.994236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.994205 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:06.995932 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:06.995914 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:07.006308 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:07.006285 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tzt\" (UniqueName: \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\") pod \"maas-api-revoke-sub-a62ee7d0-qwhkg\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:07.079950 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:07.079885 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:07.357498 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:07.357466 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg"] Nov 03 20:54:07.361481 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:54:07.361452 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad62c11_b27b_4452_b44d_d422116d798b.slice/crio-fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b WatchSource:0}: Error finding container fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b: Status 404 returned error can't find the container with id fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b Nov 03 20:54:07.602995 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:07.602958 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" event={"ID":"8ad62c11-b27b-4452-b44d-d422116d798b","Type":"ContainerStarted","Data":"fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b"} Nov 03 20:54:08.609059 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:08.609023 5954 generic.go:356] "Generic (PLEG): container finished" podID="8ad62c11-b27b-4452-b44d-d422116d798b" containerID="aeadde4b8d916fa6402fdea3ba24b67b57caa30a03fe4c6ef0e63277f5298c7d" exitCode=0 Nov 03 20:54:08.609479 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:08.609125 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" event={"ID":"8ad62c11-b27b-4452-b44d-d422116d798b","Type":"ContainerDied","Data":"aeadde4b8d916fa6402fdea3ba24b67b57caa30a03fe4c6ef0e63277f5298c7d"} Nov 03 20:54:09.752913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.752888 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:09.820894 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.820867 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\") pod \"8ad62c11-b27b-4452-b44d-d422116d798b\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " Nov 03 20:54:09.821077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.820950 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\") pod \"8ad62c11-b27b-4452-b44d-d422116d798b\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " Nov 03 20:54:09.821077 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.820991 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\" (UniqueName: \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\") pod \"8ad62c11-b27b-4452-b44d-d422116d798b\" (UID: \"8ad62c11-b27b-4452-b44d-d422116d798b\") " Nov 03 20:54:09.821246 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.821213 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca" pod "8ad62c11-b27b-4452-b44d-d422116d798b" (UID: "8ad62c11-b27b-4452-b44d-d422116d798b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:09.821332 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.821317 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:09.822904 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.822881 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt" pod "8ad62c11-b27b-4452-b44d-d422116d798b" (UID: "8ad62c11-b27b-4452-b44d-d422116d798b"). InnerVolumeSpecName "kube-api-access-w6tzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:09.823006 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.822913 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca" pod "8ad62c11-b27b-4452-b44d-d422116d798b" (UID: "8ad62c11-b27b-4452-b44d-d422116d798b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:09.922913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.922832 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad62c11-b27b-4452-b44d-d422116d798b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:09.922913 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:09.922862 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tzt\" (UniqueName: \"kubernetes.io/projected/8ad62c11-b27b-4452-b44d-d422116d798b-kube-api-access-w6tzt\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:10.619805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:10.619777 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" Nov 03 20:54:10.619805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:10.619799 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg" event={"ID":"8ad62c11-b27b-4452-b44d-d422116d798b","Type":"ContainerDied","Data":"fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b"} Nov 03 20:54:10.620038 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:10.619843 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0691753332f0122289c0f836aff78abe1575aa2c181c523223a3257089499b" Nov 03 20:54:17.736756 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:17.736717 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:54:19.864232 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:19.864196 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:28.999961 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:28.999923 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:29.000429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.000413 5954 cpu_manager.go:414] "containerMap: removing container" podUID="8ad62c11-b27b-4452-b44d-d422116d798b" containerName="revoke-keys" Nov 03 20:54:29.000477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.000430 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad62c11-b27b-4452-b44d-d422116d798b" containerName="revoke-keys" Nov 03 20:54:29.000537 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.000527 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad62c11-b27b-4452-b44d-d422116d798b" containerName="revoke-keys" Nov 03 20:54:29.004278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.004253 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.014088 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.014062 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:54:29.018688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.018648 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:54:29.018688 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.017238 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:29.018948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.017303 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:54:29.018948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.018634 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:54:29.018948 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.014401 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:54:29.020733 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.020710 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2"] Nov 03 20:54:29.023494 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.023468 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.023689 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.023659 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.023819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.023801 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.023908 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.023862 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9pr\" (UniqueName: \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.024875 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.024858 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.041277 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.041250 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2"] Nov 03 20:54:29.124812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.124770 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9pr\" (UniqueName: \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.125005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.124826 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-openshift-service-ca\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.125005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.124876 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.125005 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.124911 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9vv\" (UniqueName: \"kubernetes.io/projected/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-kube-api-access-rt9vv\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.125190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.125083 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.125190 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.125156 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-metrics-certs\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.125278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.125188 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-maas-api-tls\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.125278 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.125258 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.125710 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.125689 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.127390 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.127371 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.127482 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.127436 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.172467 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.172436 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9pr\" (UniqueName: \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\") pod \"maas-api-6c684ff7f7-ln46w\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.226195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.226157 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-metrics-certs\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.226195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.226196 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-maas-api-tls\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.226453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.226324 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-openshift-service-ca\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.226453 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.226387 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9vv\" (UniqueName: \"kubernetes.io/projected/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-kube-api-access-rt9vv\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.227154 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.227095 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-openshift-service-ca\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.228763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.228738 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-metrics-certs\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.228763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.228752 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-maas-api-tls\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.239340 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.239318 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9vv\" (UniqueName: \"kubernetes.io/projected/0eca2b89-0fb5-4c71-b9e0-0b246de84d9d-kube-api-access-rt9vv\") pod \"maas-api-78bbd54c6f-jr2s2\" (UID: \"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.321772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.321667 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:29.348934 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.348909 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:29.604962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.604936 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:29.606648 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:54:29.606614 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764205e9_1873_486a_a493_ac68671ede1b.slice/crio-75189174d97c951910bf047728c61391c8c0bd56850f129917c947be8d99729b WatchSource:0}: Error finding container 75189174d97c951910bf047728c61391c8c0bd56850f129917c947be8d99729b: Status 404 returned error can't find the container with id 75189174d97c951910bf047728c61391c8c0bd56850f129917c947be8d99729b Nov 03 20:54:29.608052 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.608034 5954 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:54:29.639409 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.639381 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2"] Nov 03 20:54:29.640719 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:54:29.640681 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eca2b89_0fb5_4c71_b9e0_0b246de84d9d.slice/crio-4179f8ff37937d8f25a15f01fd301cf41dcdaa4bfda4eeb816fd00a9182ef081 WatchSource:0}: Error finding container 4179f8ff37937d8f25a15f01fd301cf41dcdaa4bfda4eeb816fd00a9182ef081: Status 404 returned error can't find the container with id 4179f8ff37937d8f25a15f01fd301cf41dcdaa4bfda4eeb816fd00a9182ef081 Nov 03 20:54:29.730268 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.730229 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" event={"ID":"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d","Type":"ContainerStarted","Data":"4179f8ff37937d8f25a15f01fd301cf41dcdaa4bfda4eeb816fd00a9182ef081"} Nov 03 20:54:29.731584 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:29.731554 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" event={"ID":"764205e9-1873-486a-a493-ac68671ede1b","Type":"ContainerStarted","Data":"75189174d97c951910bf047728c61391c8c0bd56850f129917c947be8d99729b"} Nov 03 20:54:30.260772 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:54:30.260692 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:54:30.739207 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.739164 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" event={"ID":"764205e9-1873-486a-a493-ac68671ede1b","Type":"ContainerStarted","Data":"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1"} Nov 03 20:54:30.739406 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.739236 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:30.740762 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.740732 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" event={"ID":"0eca2b89-0fb5-4c71-b9e0-0b246de84d9d","Type":"ContainerStarted","Data":"80592534ceac5bdfd8b44d97b965f648358528f83f4cd3d9ad794f79edb60621"} Nov 03 20:54:30.740912 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.740812 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:30.760276 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.760229 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" podStartSLOduration=2.403047848 podStartE2EDuration="2.760215263s" totalImagesPullingTime="357.167415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:30.758643644 +0000 UTC m=+3411.483651245" watchObservedRunningTime="2026-11-03 20:54:30.760215263 +0000 UTC m=+3411.485222853" Nov 03 20:54:30.778784 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:30.778728 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" podStartSLOduration=2.337881286 podStartE2EDuration="2.778712642s" totalImagesPullingTime="440.831356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:30.777831299 +0000 UTC m=+3411.502838898" watchObservedRunningTime="2026-11-03 20:54:30.778712642 +0000 UTC m=+3411.503720239" Nov 03 20:54:36.015130 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.015057 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:36.504251 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.504218 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:36.751291 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.751257 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:36.751626 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.751606 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-jr2s2" Nov 03 20:54:36.822881 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.822793 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:36.829384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.829356 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:36.829556 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:36.829535 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" podUID="764205e9-1873-486a-a493-ac68671ede1b" containerName="maas-api" containerID="cri-o://572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1" gracePeriod=30 Nov 03 20:54:37.085893 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.085832 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:37.208496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.208464 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\") pod \"764205e9-1873-486a-a493-ac68671ede1b\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " Nov 03 20:54:37.208671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.208518 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\" (UniqueName: \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\") pod \"764205e9-1873-486a-a493-ac68671ede1b\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " Nov 03 20:54:37.208671 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.208633 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\") pod \"764205e9-1873-486a-a493-ac68671ede1b\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " Nov 03 20:54:37.208786 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.208711 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\") pod \"764205e9-1873-486a-a493-ac68671ede1b\" (UID: \"764205e9-1873-486a-a493-ac68671ede1b\") " Nov 03 20:54:37.209165 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.209102 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca" pod "764205e9-1873-486a-a493-ac68671ede1b" (UID: "764205e9-1873-486a-a493-ac68671ede1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:37.210607 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.210576 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs" pod "764205e9-1873-486a-a493-ac68671ede1b" (UID: "764205e9-1873-486a-a493-ac68671ede1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:37.210719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.210634 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr" pod "764205e9-1873-486a-a493-ac68671ede1b" (UID: "764205e9-1873-486a-a493-ac68671ede1b"). InnerVolumeSpecName "kube-api-access-7k9pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:37.210719 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.210701 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls" pod "764205e9-1873-486a-a493-ac68671ede1b" (UID: "764205e9-1873-486a-a493-ac68671ede1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:37.310515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.310486 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:37.310515 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.310512 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/764205e9-1873-486a-a493-ac68671ede1b-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:37.310711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.310523 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/764205e9-1873-486a-a493-ac68671ede1b-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:37.310711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.310533 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k9pr\" (UniqueName: \"kubernetes.io/projected/764205e9-1873-486a-a493-ac68671ede1b-kube-api-access-7k9pr\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:37.778867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.778781 5954 generic.go:356] "Generic (PLEG): container finished" podID="764205e9-1873-486a-a493-ac68671ede1b" containerID="572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1" exitCode=0 Nov 03 20:54:37.778867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.778837 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" Nov 03 20:54:37.779128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.778870 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" event={"ID":"764205e9-1873-486a-a493-ac68671ede1b","Type":"ContainerDied","Data":"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1"} Nov 03 20:54:37.779128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.778928 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w" event={"ID":"764205e9-1873-486a-a493-ac68671ede1b","Type":"ContainerDied","Data":"75189174d97c951910bf047728c61391c8c0bd56850f129917c947be8d99729b"} Nov 03 20:54:37.779128 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.778952 5954 scope.go:122] "RemoveContainer" containerID="572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1" Nov 03 20:54:37.788846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.788831 5954 scope.go:122] "RemoveContainer" containerID="572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1" Nov 03 20:54:37.789138 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:54:37.789120 5954 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1\": container with ID starting with 572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1 not found: ID does not exist" containerID="572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1" Nov 03 20:54:37.789195 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.789148 5954 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1"} err="failed to get container status \"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1\": rpc error: code = NotFound desc = could not find container \"572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1\": container with ID starting with 572e73a5dc33785be64e82360c0b8921f3064f908c19053e7ace25c38fa258b1 not found: ID does not exist" Nov 03 20:54:37.797213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.797188 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:37.801650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:37.801631 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c684ff7f7-ln46w"] Nov 03 20:54:39.405504 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:39.405468 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764205e9-1873-486a-a493-ac68671ede1b" path="/var/lib/kubelet/pods/764205e9-1873-486a-a493-ac68671ede1b/volumes" Nov 03 20:54:39.625635 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:39.625600 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:39.753414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:39.753338 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:40.804613 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.804569 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:54:40.805090 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.805075 5954 cpu_manager.go:414] "containerMap: removing container" podUID="764205e9-1873-486a-a493-ac68671ede1b" containerName="maas-api" Nov 03 20:54:40.805168 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.805092 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764205e9-1873-486a-a493-ac68671ede1b" containerName="maas-api" Nov 03 20:54:40.805208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.805182 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="764205e9-1873-486a-a493-ac68671ede1b" containerName="maas-api" Nov 03 20:54:40.810477 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.810447 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:40.816095 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.816071 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bdf98686-serving-cert\"" Nov 03 20:54:40.822703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.822677 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:54:40.837806 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.837773 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:40.949630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.949593 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5d9\" (UniqueName: \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:40.949630 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.949635 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:40.949874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.949665 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:40.949874 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:40.949803 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.051378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.051343 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.051575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.051437 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5d9\" (UniqueName: \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.051575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.051459 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.051575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.051489 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.052258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.052228 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.054413 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.054376 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.054658 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.054515 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.061208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.061180 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:41.100619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.100579 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5d9\" (UniqueName: \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\") pod \"maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.128258 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.128224 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:41.185929 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.185605 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:41.410012 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.409985 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:54:41.412394 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:54:41.412366 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fff2ea4_0ae8_4d4e_bc90_3b412cf7207f.slice/crio-bc884e48089e65d4caa31a1dfeddae790b4ed0362967eb275aab981a3e9a7d88 WatchSource:0}: Error finding container bc884e48089e65d4caa31a1dfeddae790b4ed0362967eb275aab981a3e9a7d88: Status 404 returned error can't find the container with id bc884e48089e65d4caa31a1dfeddae790b4ed0362967eb275aab981a3e9a7d88 Nov 03 20:54:41.804029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.803947 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" event={"ID":"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f","Type":"ContainerStarted","Data":"bc884e48089e65d4caa31a1dfeddae790b4ed0362967eb275aab981a3e9a7d88"} Nov 03 20:54:41.854792 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:41.854752 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:42.559586 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:42.559554 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:42.811840 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:42.811747 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" event={"ID":"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f","Type":"ContainerStarted","Data":"d7900e9a024af6a86d482aeab34e659d8a42ef19da3770fe5788cde568dbb86f"} Nov 03 20:54:42.812020 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:42.811860 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:42.833930 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:42.833881 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" podStartSLOduration=2.430150935 podStartE2EDuration="2.833867195s" totalImagesPullingTime="403.71626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:42.831670557 +0000 UTC m=+3423.556678155" watchObservedRunningTime="2026-11-03 20:54:42.833867195 +0000 UTC m=+3423.558874790" Nov 03 20:54:42.865763 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:42.865734 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:54:43.347530 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:43.347501 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:44.050670 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:44.050640 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:44.733210 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:44.733172 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:45.442838 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:45.442806 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:46.148850 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:46.148803 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:46.842460 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:46.842425 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:47.546989 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:47.546955 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:48.341623 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:48.341583 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:48.822083 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:48.822055 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:54:49.038078 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:49.038047 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:49.798734 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:49.798697 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:50.041395 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:50.041316 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:50.509280 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:50.509236 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:51.444348 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:51.444315 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:52.155129 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:52.155074 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:52.852396 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:52.852362 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:53.547265 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:53.547232 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:54.246305 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:54.246265 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:54.946135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:54.946078 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:55.815404 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:55.815372 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:56.643574 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:56.643539 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:57.345551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:57.345518 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:58.044692 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:58.044651 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:58.846652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:58.846618 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:59.545598 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:54:59.545562 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:00.249001 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:00.248960 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:00.258261 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:55:00.258225 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:55:00.942540 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:00.942499 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:01.725158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:01.725102 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.404387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:02.404354 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.041100 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:03.041060 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.745065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:03.745032 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:04.458324 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:04.458288 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.140696 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.140654 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.629729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.629691 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.679911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.679880 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:55:05.680167 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.680090 5954 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" podUID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" containerName="maas-api" containerID="cri-o://d7900e9a024af6a86d482aeab34e659d8a42ef19da3770fe5788cde568dbb86f" gracePeriod=30 Nov 03 20:55:05.730369 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.730324 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.830790 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.830760 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.946516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.946477 5954 generic.go:356] "Generic (PLEG): container finished" podID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" containerID="d7900e9a024af6a86d482aeab34e659d8a42ef19da3770fe5788cde568dbb86f" exitCode=0 Nov 03 20:55:05.946711 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.946674 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" event={"ID":"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f","Type":"ContainerDied","Data":"d7900e9a024af6a86d482aeab34e659d8a42ef19da3770fe5788cde568dbb86f"} Nov 03 20:55:05.962978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:05.962956 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:55:06.023029 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.022993 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\" (UniqueName: \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\") pod \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " Nov 03 20:55:06.023248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.023044 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\") pod \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " Nov 03 20:55:06.023248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.023127 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\") pod \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " Nov 03 20:55:06.023393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.023238 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\") pod \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\" (UID: \"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f\") " Nov 03 20:55:06.023634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.023605 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca" pod "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" (UID: "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:06.023771 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.023733 5954 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-openshift-service-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.025304 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.025278 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls" pod "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" (UID: "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:06.025407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.025358 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs" pod "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" (UID: "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:06.025407 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.025385 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9" pod "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" (UID: "6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f"). InnerVolumeSpecName "kube-api-access-tw5d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:06.124344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.124307 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5d9\" (UniqueName: \"kubernetes.io/projected/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-kube-api-access-tw5d9\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.124344 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.124342 5954 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-metrics-certs\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.124572 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.124357 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f-maas-api-tls\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.498262 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.498224 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:06.596375 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.596336 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:06.953293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.953258 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" Nov 03 20:55:06.953815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.953268 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp" event={"ID":"6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f","Type":"ContainerDied","Data":"bc884e48089e65d4caa31a1dfeddae790b4ed0362967eb275aab981a3e9a7d88"} Nov 03 20:55:06.953815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:06.953399 5954 scope.go:122] "RemoveContainer" containerID="d7900e9a024af6a86d482aeab34e659d8a42ef19da3770fe5788cde568dbb86f" Nov 03 20:55:07.041891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:07.041851 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:55:07.065956 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:07.065924 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bdf98686-5c568f8b6b-hmhcp"] Nov 03 20:55:07.405884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:07.405851 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" path="/var/lib/kubelet/pods/6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f/volumes" Nov 03 20:55:07.661701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:07.661614 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:55:11.879323 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:11.879247 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.123480 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:13.123433 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.576759 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:13.576660 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.782993 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:13.782940 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:16.559236 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:16.559200 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:16.747809 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:16.747766 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:17.693846 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:17.693804 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:17.809153 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:17.809119 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:17.912093 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:17.912062 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:18.665056 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:18.665020 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:19.456073 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:19.456041 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:19.975261 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:19.975222 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:55:20.155757 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:20.155718 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:21.000812 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:21.000776 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:21.643368 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:21.643324 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:22.350649 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:22.350609 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:23.045085 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:23.045040 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:23.743609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:23.743572 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:24.547378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:24.547332 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:25.241618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:25.241587 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:26.088926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:26.088887 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:26.872827 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:26.872789 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:27.613813 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:27.613749 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:28.262582 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:28.262548 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:29.015213 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:29.015180 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:29.660031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:29.660000 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:30.257939 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:55:30.257899 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:55:30.358346 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:30.358308 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.050219 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:31.050169 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.509248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:31.509217 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.790631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:31.790535 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:32.453352 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:32.453321 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:33.242541 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:33.242502 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:33.949281 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:33.949249 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:34.647270 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:34.647234 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:35.352244 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:35.352208 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:36.051815 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:36.051782 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:36.756941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:36.756911 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:37.551433 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:37.551401 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:38.324616 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:38.324573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:38.954981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:38.954949 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:39.648248 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:39.648212 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:40.458752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:40.458719 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:41.155301 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:41.155253 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:41.953384 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:41.953336 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:42.648680 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:42.648638 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:43.346905 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:43.346861 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:44.055188 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:44.055151 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:44.753049 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:44.753010 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.550151 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:45.550099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.722169 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:45.722136 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.822133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:45.822027 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:46.238328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:46.238292 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:46.971137 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:46.971083 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:47.661133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:47.661076 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:48.021988 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:48.021903 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4"] Nov 03 20:55:48.024737 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:48.024710 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-2gzq4"] Nov 03 20:55:48.123646 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:48.123615 5954 scope.go:122] "RemoveContainer" containerID="066c3a5f5de4e2fa5b1415ab1b37cb0789fe8a1d368d1f4d9fdde421091f16ca" Nov 03 20:55:48.354729 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:48.354696 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:49.045844 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:49.045809 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:49.403566 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:49.403534 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91de4c5b-f078-410f-b55a-3705fe52c944" path="/var/lib/kubelet/pods/91de4c5b-f078-410f-b55a-3705fe52c944/volumes" Nov 03 20:55:49.753678 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:49.753591 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:50.092618 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:50.092532 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:50.548528 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:50.548492 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:51.259473 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:51.259437 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:52.051241 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:52.051206 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:52.738383 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:52.738346 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:53.445370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:53.445333 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:54.145857 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:54.145821 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:54.942068 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:54.942030 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:55.651442 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:55.651401 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:56.353859 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:56.353818 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:57.047606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:57.047573 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:57.779229 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:57.779190 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:58.451941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:58.451907 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:59.263048 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:59.263001 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:59.948030 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:55:59.947995 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:00.258920 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:56:00.258834 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:56:00.650780 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:00.650745 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:01.352037 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:01.352006 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:02.047643 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:02.047608 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:02.956999 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:02.956952 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:03.650765 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:03.650729 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:04.348476 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:04.348441 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:05.055911 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:05.055880 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:05.743695 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:05.743659 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:06.465256 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:06.465220 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:07.021414 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:07.021378 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl"] Nov 03 20:56:07.024171 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:07.024142 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2pzpl"] Nov 03 20:56:07.224552 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:07.224522 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:07.403575 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:07.403542 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4999a164-7215-4eb5-9c6e-b5a53bc76b69" path="/var/lib/kubelet/pods/4999a164-7215-4eb5-9c6e-b5a53bc76b69/volumes" Nov 03 20:56:07.950364 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:07.950332 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:08.645135 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:08.645081 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:09.351634 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:09.351590 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:10.052700 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:10.052657 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:10.854926 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:10.854893 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:11.544435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:11.544395 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:12.244064 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:12.244029 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:12.936362 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:12.936328 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:13.639222 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:13.639186 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:14.340490 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:14.340453 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:15.041715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:15.041661 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:15.745133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:15.745072 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:16.506033 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:16.505906 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:17.212327 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:17.212292 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:17.843279 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:17.843246 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:18.542555 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:18.542521 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:19.252496 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:19.252467 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:19.936065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:19.936032 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:20.644597 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:20.644553 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:21.335419 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:21.335384 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:22.138457 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:22.138427 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:22.834969 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:22.834927 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:23.539714 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:23.539678 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:24.240469 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:24.240438 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:25.035978 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:25.035935 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:25.738393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:25.738357 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:26.443726 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:26.443624 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:27.139619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:27.139582 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:27.845679 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:27.845646 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:28.546701 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:28.546662 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:29.248526 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:29.248492 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:29.961604 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:29.961546 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:30.257314 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:56:30.257232 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:56:30.753385 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:30.753351 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:31.453161 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:31.453099 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:32.155548 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:32.155512 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:32.853808 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:32.853777 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:33.533522 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:33.533487 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:34.235072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:34.235033 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:34.935779 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:34.935737 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:35.631415 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:35.631384 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:36.391205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:36.391169 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:37.031869 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:37.031834 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:37.743933 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:37.743899 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:38.545657 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:38.545623 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:39.247031 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:39.246997 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:40.041293 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:40.041258 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:40.734143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:40.734087 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:41.438907 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:41.438874 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:42.134819 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:42.134786 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:42.836860 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:42.836826 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:43.540752 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:43.540719 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:44.247631 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:44.247598 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:44.969879 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:44.969847 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:45.637158 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:45.637125 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:46.339525 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:46.339489 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:47.032133 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:47.032088 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:47.743435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:47.743095 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:48.153178 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:48.153151 5954 scope.go:122] "RemoveContainer" containerID="0bca03cc46cd0e97d2953132b0851b0d676ab7d72a3d6e4930e4d391aadfa675" Nov 03 20:56:48.441652 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:48.441570 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:49.135018 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:49.134986 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:49.833977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:49.833948 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:50.148876 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:50.148846 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:50.532387 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:50.532301 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:51.240008 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.239976 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:51.665481 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.665449 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv"] Nov 03 20:56:51.665909 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.665894 5954 cpu_manager.go:414] "containerMap: removing container" podUID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" containerName="maas-api" Nov 03 20:56:51.665977 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.665909 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" containerName="maas-api" Nov 03 20:56:51.666017 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.665989 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fff2ea4-0ae8-4d4e-bc90-3b412cf7207f" containerName="maas-api" Nov 03 20:56:51.670447 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.670428 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.676962 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.676938 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:51.677091 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.677038 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xhgp6\"" Nov 03 20:56:51.678462 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.678438 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv"] Nov 03 20:56:51.753728 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.753692 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.753915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.753805 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.753915 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.753883 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98lh\" (UniqueName: \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.854773 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.854735 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.854968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.854807 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.854968 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.854856 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98lh\" (UniqueName: \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.855401 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.855381 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.857166 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.857142 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.867084 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.867059 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98lh\" (UniqueName: \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:51.985941 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:51.985853 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:52.096891 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.096857 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:52.267386 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.267344 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv"] Nov 03 20:56:52.270587 ip-10-0-135-70 kubenswrapper[5954]: W1103 20:56:52.270557 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1605d1_4c26_4d4c_a03a_d1c03a4cdc3b.slice/crio-41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902 WatchSource:0}: Error finding container 41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902: Status 404 returned error can't find the container with id 41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902 Nov 03 20:56:52.528209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.528101 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" event={"ID":"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b","Type":"ContainerStarted","Data":"cea5eb113961099e73b0ef09e4a59bcbe82c01285092181393d170bf495fa2a7"} Nov 03 20:56:52.528209 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.528185 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" event={"ID":"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b","Type":"ContainerStarted","Data":"41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902"} Nov 03 20:56:52.546202 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.546152 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" podStartSLOduration=1.546137402 podStartE2EDuration="1.546137402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:52.545618823 +0000 UTC m=+3553.270626420" watchObservedRunningTime="2026-11-03 20:56:52.546137402 +0000 UTC m=+3553.271144995" Nov 03 20:56:52.835693 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:52.835597 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:53.535356 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:53.535320 5954 generic.go:356] "Generic (PLEG): container finished" podID="0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" containerID="cea5eb113961099e73b0ef09e4a59bcbe82c01285092181393d170bf495fa2a7" exitCode=0 Nov 03 20:56:53.535767 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:53.535410 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" event={"ID":"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b","Type":"ContainerDied","Data":"cea5eb113961099e73b0ef09e4a59bcbe82c01285092181393d170bf495fa2a7"} Nov 03 20:56:53.567959 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:53.567925 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:54.340661 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.340604 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:54.687410 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.687379 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:54.802970 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.802940 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\") pod \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " Nov 03 20:56:54.803208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.802997 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\" (UniqueName: \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\") pod \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " Nov 03 20:56:54.803208 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.803077 5954 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\") pod \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\" (UID: \"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b\") " Nov 03 20:56:54.803516 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.803494 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca" pod "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" (UID: "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:54.805066 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.805032 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca" pod "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" (UID: "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:54.805198 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.805070 5954 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh" pod "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" (UID: "0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b"). InnerVolumeSpecName "kube-api-access-s98lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:54.904772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.904734 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-xks-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:54.904772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.904768 5954 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98lh\" (UniqueName: \"kubernetes.io/projected/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-kube-api-access-s98lh\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:54.904772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:54.904779 5954 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b-maas-api-openshift-ca\") on node \"ip-10-0-135-70.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:55.205311 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:55.205194 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:55.547705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:55.547613 5954 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" Nov 03 20:56:55.547705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:55.547648 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv" event={"ID":"0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b","Type":"ContainerDied","Data":"41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902"} Nov 03 20:56:55.547705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:55.547703 5954 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41961378b22162f0dc85f21003cfbf86e6bf883af8aaa6dfc829ea8c2420d902" Nov 03 20:56:55.857461 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:55.857433 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:56.553361 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:56.553314 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:57.246705 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:57.246647 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:57.950702 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:57.950637 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:58.649551 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:58.649508 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:59.343513 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:56:59.343475 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.050289 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:00.050257 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.258597 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:57:00.258563 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:57:00.750341 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:00.750304 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:01.454047 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:01.454014 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:01.859573 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:01.859532 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:01.969394 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:01.969363 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:02.175034 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:02.174952 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:07.763337 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:07.763279 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:07.940424 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:07.940386 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv"] Nov 03 20:57:07.942650 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:07.942617 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:07.942799 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:07.942649 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0fe875-smwxv"] Nov 03 20:57:08.037787 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:08.037655 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:08.507684 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:08.507649 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:09.373299 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:09.373263 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:57:09.403856 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:09.403825 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" path="/var/lib/kubelet/pods/0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b/volumes" Nov 03 20:57:14.824489 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:14.824456 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:14.954143 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:14.954090 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:15.176116 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:15.176067 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:17.254380 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:17.254343 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:57:19.792296 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:19.792253 5954 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-v6ll6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:57:19.792822 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:19.792333 5954 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-v6ll6" podUID="9142de3d-195d-40ca-92b6-6777196e15d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:57:22.652955 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:22.652918 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:57:30.258184 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:57:30.258146 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:57:37.867731 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:37.867692 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:57:47.898789 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:47.898740 5954 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal" podUID="d161bc542803fe55ee71704a451b4d4d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Nov 03 20:57:48.174715 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:48.174638 5954 scope.go:122] "RemoveContainer" containerID="e94f564f5520c8099e0ac3c691a97ee944240520ef8098392dda183a30ee04fb" Nov 03 20:57:50.207703 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:57:50.207626 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:00.256871 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:58:00.256831 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:58:04.567772 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:04.567733 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-spzh9"] Nov 03 20:58:12.997393 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:12.997345 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:13.100946 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:13.100908 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:15.649065 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:15.649025 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7878f95ccc-l27rs_a6709e2e-e552-4789-ad94-0788a9583418/manager/0.log" Nov 03 20:58:15.841046 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:15.841014 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hvd4x_71d10093-69e4-4465-bc5c-4685a95d9427/manager/0.log" Nov 03 20:58:16.529252 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:16.529223 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 20:58:16.931554 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:16.931522 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lv9j5_694e6ee4-c04f-48ca-894f-1b0c4a63a7b5/manager/0.log" Nov 03 20:58:17.060581 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:17.060548 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:17.919981 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:17.919947 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78bbd54c6f-jr2s2_0eca2b89-0fb5-4c71-b9e0-0b246de84d9d/maas-api/0.log" Nov 03 20:58:18.482867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:18.482836 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-g8xd4_a2bb1232-ea47-4c42-b06c-7b2b62f4719c/revoke-keys/0.log" Nov 03 20:58:18.660174 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:18.660147 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-m6dx8_84d590b1-2bab-4faf-99f8-a1de6503eca3/revoke-keys/0.log" Nov 03 20:58:19.013647 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:19.013618 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a62ee7d0-qwhkg_8ad62c11-b27b-4452-b44d-d422116d798b/revoke-keys/0.log" Nov 03 20:58:21.281609 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:21.281575 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-drz9b_1bd00c68-f651-453e-b2e9-126a86d5ba11/manager/0.log" Nov 03 20:58:21.465255 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:21.465228 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5nlk2_1e5bc593-5e89-4d37-8539-b96830cae8a6/kuadrant-console-plugin/0.log" Nov 03 20:58:21.877606 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:21.877577 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-spzh9_1f257ac6-b47e-4fa8-80cd-66c2e263b816/limitador/0.log" Nov 03 20:58:23.057081 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:23.057050 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bzp6h_f29ac0d4-d574-4773-8d82-81f822379713/discovery/0.log" Nov 03 20:58:23.426619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:23.426588 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-vncsb_6e045ad4-d958-4322-814b-d2c57c8d8cfd/kube-auth-proxy/0.log" Nov 03 20:58:24.202553 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:24.202525 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-mf6wt_ddad7abe-544d-4e16-a193-f7c905fe1e64/istio-proxy/0.log" Nov 03 20:58:25.121370 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:25.121339 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-v6ll6_9142de3d-195d-40ca-92b6-6777196e15d5/istio-proxy/0.log" Nov 03 20:58:25.302898 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:25.302857 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zdxwl_0fb68c68-b69e-4e9b-bc3f-589b465a1ffa/istio-proxy/0.log" Nov 03 20:58:25.672072 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:25.672045 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h4h49_3126a1b2-1b19-441d-912d-7c4b4701b779/payload-pre-processing/0.log" Nov 03 20:58:27.751982 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:27.751951 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl_930811f6-8cc8-4139-900f-7b4383999138/main/0.log" Nov 03 20:58:30.257437 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:58:30.257400 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:58:36.023328 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:36.023293 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8"] Nov 03 20:58:36.026150 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:36.026125 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-m6dx8"] Nov 03 20:58:37.405429 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:37.405397 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d590b1-2bab-4faf-99f8-a1de6503eca3" path="/var/lib/kubelet/pods/84d590b1-2bab-4faf-99f8-a1de6503eca3/volumes" Nov 03 20:58:38.467919 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:38.467886 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7878f95ccc-l27rs_a6709e2e-e552-4789-ad94-0788a9583418/manager/0.log" Nov 03 20:58:38.687486 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:38.687460 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-hvd4x_71d10093-69e4-4465-bc5c-4685a95d9427/manager/0.log" Nov 03 20:58:39.317835 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:39.317802 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 20:58:39.712884 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:39.712853 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-lv9j5_694e6ee4-c04f-48ca-894f-1b0c4a63a7b5/manager/0.log" Nov 03 20:58:40.474511 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:40.474480 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78bbd54c6f-jr2s2_0eca2b89-0fb5-4c71-b9e0-0b246de84d9d/maas-api/0.log" Nov 03 20:58:41.029523 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:41.029494 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-g8xd4_a2bb1232-ea47-4c42-b06c-7b2b62f4719c/revoke-keys/0.log" Nov 03 20:58:41.400951 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:41.400915 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a62ee7d0-qwhkg_8ad62c11-b27b-4452-b44d-d422116d798b/revoke-keys/0.log" Nov 03 20:58:43.373632 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:43.373598 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-drz9b_1bd00c68-f651-453e-b2e9-126a86d5ba11/manager/0.log" Nov 03 20:58:43.559681 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:43.559648 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5nlk2_1e5bc593-5e89-4d37-8539-b96830cae8a6/kuadrant-console-plugin/0.log" Nov 03 20:58:43.944358 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:43.944330 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-spzh9_1f257ac6-b47e-4fa8-80cd-66c2e263b816/limitador/0.log" Nov 03 20:58:44.930360 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:44.930328 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bzp6h_f29ac0d4-d574-4773-8d82-81f822379713/discovery/0.log" Nov 03 20:58:45.311589 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:45.311507 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-vncsb_6e045ad4-d958-4322-814b-d2c57c8d8cfd/kube-auth-proxy/0.log" Nov 03 20:58:46.063868 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:46.063836 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-mf6wt_ddad7abe-544d-4e16-a193-f7c905fe1e64/istio-proxy/0.log" Nov 03 20:58:47.041379 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:47.041345 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-v6ll6_9142de3d-195d-40ca-92b6-6777196e15d5/istio-proxy/0.log" Nov 03 20:58:47.240867 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:47.240830 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zdxwl_0fb68c68-b69e-4e9b-bc3f-589b465a1ffa/istio-proxy/0.log" Nov 03 20:58:47.610619 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:47.610586 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h4h49_3126a1b2-1b19-441d-912d-7c4b4701b779/payload-pre-processing/0.log" Nov 03 20:58:49.022250 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:49.022212 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4"] Nov 03 20:58:49.024805 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:49.024776 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-g8xd4"] Nov 03 20:58:49.240938 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:49.240907 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8grxtl_930811f6-8cc8-4139-900f-7b4383999138/main/0.log" Nov 03 20:58:49.403535 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:49.403500 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bb1232-ea47-4c42-b06c-7b2b62f4719c" path="/var/lib/kubelet/pods/a2bb1232-ea47-4c42-b06c-7b2b62f4719c/volumes" Nov 03 20:58:50.238228 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:50.238195 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:53.196205 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:58:53.196170 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:00.257989 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:59:00.257950 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:59:10.021160 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:10.021121 5954 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg"] Nov 03 20:59:10.023896 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:10.023871 5954 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62ee7d0-qwhkg"] Nov 03 20:59:11.403435 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:11.403401 5954 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad62c11-b27b-4452-b44d-d422116d798b" path="/var/lib/kubelet/pods/8ad62c11-b27b-4452-b44d-d422116d798b/volumes" Nov 03 20:59:15.583378 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:15.583340 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:15.683465 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:15.683431 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:29.314625 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:29.314539 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 20:59:29.602834 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:29.602808 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 20:59:30.257849 ip-10-0-135-70 kubenswrapper[5954]: E1103 20:59:30.257811 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 20:59:48.205436 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:48.205321 5954 scope.go:122] "RemoveContainer" containerID="278525ab3fed5f06d1a73ca82b44290425b9df49ce5e609eaceef1bdf32da384" Nov 03 20:59:48.214748 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:48.214726 5954 scope.go:122] "RemoveContainer" containerID="a1ed98ee69cc06b52f74b71342a9c0a886be442816b0adce98bb3fc5e9841506" Nov 03 20:59:50.310751 ip-10-0-135-70 kubenswrapper[5954]: I1103 20:59:50.310714 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:00.258116 ip-10-0-135-70 kubenswrapper[5954]: E1103 21:00:00.258065 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 21:00:09.729249 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:09.729209 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:09.829444 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:09.829411 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:29.493451 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:29.493414 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 21:00:29.772769 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:29.772668 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6dd9f77dd9-sxcbm_16c22b88-db5d-4e5f-8ada-5bfb6c7dd972/manager/0.log" Nov 03 21:00:30.257834 ip-10-0-135-70 kubenswrapper[5954]: E1103 21:00:30.257801 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor" Nov 03 21:00:31.052940 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:31.052899 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.150735 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:31.150697 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.608525 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:34.608496 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lb594_18e97b6a-be41-42af-90ec-bd56e6d791d6/global-pull-secret-syncer/0.log" Nov 03 21:00:34.817825 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:34.817768 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" Nov 03 21:00:34.961684 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:34.961599 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-70.us-east-2.compute.internal_d161bc542803fe55ee71704a451b4d4d/haproxy/0.log" Nov 03 21:00:43.870973 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.870937 5954 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc"] Nov 03 21:00:43.871689 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.871663 5954 cpu_manager.go:414] "containerMap: removing container" podUID="0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" containerName="revoke-keys" Nov 03 21:00:43.871689 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.871685 5954 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" containerName="revoke-keys" Nov 03 21:00:43.871915 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.871798 5954 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1605d1-4c26-4d4c-a03a-d1c03a4cdc3b" containerName="revoke-keys" Nov 03 21:00:43.875286 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.875264 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:43.881051 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.881028 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"openshift-service-ca.crt\"" Nov 03 21:00:43.881305 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.881283 5954 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"kube-root-ca.crt\"" Nov 03 21:00:43.882346 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.882322 5954 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpp9v\"/\"default-dockercfg-dsb8v\"" Nov 03 21:00:43.883239 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.883213 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc"] Nov 03 21:00:43.924639 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.924609 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-drz9b_1bd00c68-f651-453e-b2e9-126a86d5ba11/manager/0.log" Nov 03 21:00:43.957296 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.957271 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5nlk2_1e5bc593-5e89-4d37-8539-b96830cae8a6/kuadrant-console-plugin/0.log" Nov 03 21:00:43.988281 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.988244 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-lib-modules\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:43.988459 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.988289 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-podres\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:43.988459 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.988375 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqpzf\" (UniqueName: \"kubernetes.io/projected/ffa253d1-79e6-4cfd-8c75-ac486a036e81-kube-api-access-pqpzf\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:43.988459 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.988431 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-proc\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:43.988573 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:43.988502 5954 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-sys\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.044072 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.044040 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-spzh9_1f257ac6-b47e-4fa8-80cd-66c2e263b816/limitador/0.log" Nov 03 21:00:44.089833 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.089795 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-sys\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.089877 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-lib-modules\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.089917 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-sys\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.089922 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-podres\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.089977 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-podres\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.090002 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-lib-modules\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090038 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.090039 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqpzf\" (UniqueName: \"kubernetes.io/projected/ffa253d1-79e6-4cfd-8c75-ac486a036e81-kube-api-access-pqpzf\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090340 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.090067 5954 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-proc\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.090340 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.090161 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ffa253d1-79e6-4cfd-8c75-ac486a036e81-proc\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.102102 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.102079 5954 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqpzf\" (UniqueName: \"kubernetes.io/projected/ffa253d1-79e6-4cfd-8c75-ac486a036e81-kube-api-access-pqpzf\") pod \"perf-node-gather-daemonset-lpczc\" (UID: \"ffa253d1-79e6-4cfd-8c75-ac486a036e81\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.189679 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.189578 5954 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.465719 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.465629 5954 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc"] Nov 03 21:00:44.468807 ip-10-0-135-70 kubenswrapper[5954]: W1103 21:00:44.468775 5954 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podffa253d1_79e6_4cfd_8c75_ac486a036e81.slice/crio-a7461c5fb4648bbc1e4af7e59e691e47a720d055425bf2b374596146e49fc2f3 WatchSource:0}: Error finding container a7461c5fb4648bbc1e4af7e59e691e47a720d055425bf2b374596146e49fc2f3: Status 404 returned error can't find the container with id a7461c5fb4648bbc1e4af7e59e691e47a720d055425bf2b374596146e49fc2f3 Nov 03 21:00:44.782087 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.781990 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" event={"ID":"ffa253d1-79e6-4cfd-8c75-ac486a036e81","Type":"ContainerStarted","Data":"7eccb4f57d4466021f0aaae26728919767fe5baf7f64f7b5c413e9d5cdd5b224"} Nov 03 21:00:44.782087 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.782051 5954 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" event={"ID":"ffa253d1-79e6-4cfd-8c75-ac486a036e81","Type":"ContainerStarted","Data":"a7461c5fb4648bbc1e4af7e59e691e47a720d055425bf2b374596146e49fc2f3"} Nov 03 21:00:44.782327 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.782127 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:44.801019 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:44.800964 5954 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" podStartSLOduration=1.800950119 podStartE2EDuration="1.800950119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:44.799589696 +0000 UTC m=+3785.524597292" watchObservedRunningTime="2026-11-03 21:00:44.800950119 +0000 UTC m=+3785.525957715" Nov 03 21:00:45.117995 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.117964 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/alertmanager/0.log" Nov 03 21:00:45.159444 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.159420 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/config-reloader/0.log" Nov 03 21:00:45.187368 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.187333 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/kube-rbac-proxy-web/0.log" Nov 03 21:00:45.212882 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.212853 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/kube-rbac-proxy/0.log" Nov 03 21:00:45.240270 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.240245 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/kube-rbac-proxy-metric/0.log" Nov 03 21:00:45.265740 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.265712 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/prom-label-proxy/0.log" Nov 03 21:00:45.291849 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.291826 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_44071c1c-bee7-426f-820d-436d709abcc2/init-config-reloader/0.log" Nov 03 21:00:45.376292 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.376211 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5slgw_31e1076f-26b7-40eb-a11f-b9eca41ee6fc/kube-state-metrics/0.log" Nov 03 21:00:45.399685 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.399653 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5slgw_31e1076f-26b7-40eb-a11f-b9eca41ee6fc/kube-rbac-proxy-main/0.log" Nov 03 21:00:45.426768 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.426741 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5slgw_31e1076f-26b7-40eb-a11f-b9eca41ee6fc/kube-rbac-proxy-self/0.log" Nov 03 21:00:45.459530 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.459500 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5687c6f496-d89k4_78dccc20-8871-4155-aa48-7ffbf940209f/metrics-server/0.log" Nov 03 21:00:45.488052 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.488027 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-52pfh_73f24aac-5672-4b00-b199-324e3da68852/monitoring-plugin/0.log" Nov 03 21:00:45.608421 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.608389 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7sbn_426ca1a6-344a-4f33-9e58-50be798ed0a9/node-exporter/0.log" Nov 03 21:00:45.633553 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.633473 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7sbn_426ca1a6-344a-4f33-9e58-50be798ed0a9/kube-rbac-proxy/0.log" Nov 03 21:00:45.658536 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.658510 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7sbn_426ca1a6-344a-4f33-9e58-50be798ed0a9/init-textfile/0.log" Nov 03 21:00:45.778982 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.778953 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7cv4w_fc3cd7f2-0667-4487-965c-44e36b07ceb3/kube-rbac-proxy-main/0.log" Nov 03 21:00:45.804995 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.804961 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7cv4w_fc3cd7f2-0667-4487-965c-44e36b07ceb3/kube-rbac-proxy-self/0.log" Nov 03 21:00:45.831403 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.831372 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-7cv4w_fc3cd7f2-0667-4487-965c-44e36b07ceb3/openshift-state-metrics/0.log" Nov 03 21:00:45.872655 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.872629 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/prometheus/0.log" Nov 03 21:00:45.898666 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.898598 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/config-reloader/0.log" Nov 03 21:00:45.924950 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.924918 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/thanos-sidecar/0.log" Nov 03 21:00:45.950367 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.950340 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/kube-rbac-proxy-web/0.log" Nov 03 21:00:45.980439 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:45.980409 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/kube-rbac-proxy/0.log" Nov 03 21:00:46.007689 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.007658 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/kube-rbac-proxy-thanos/0.log" Nov 03 21:00:46.035930 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.035904 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c37eff7-3832-4dc4-a825-9d7533ca0db4/init-config-reloader/0.log" Nov 03 21:00:46.072310 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.072279 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6vrqs_fc34dba5-14d2-402c-99ed-0a27e607616d/prometheus-operator/0.log" Nov 03 21:00:46.094844 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.094817 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6vrqs_fc34dba5-14d2-402c-99ed-0a27e607616d/kube-rbac-proxy/0.log" Nov 03 21:00:46.159594 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.159514 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbb5848c7-wcspz_b769fedf-c73d-4aad-90dc-a8a49d0693dc/telemeter-client/0.log" Nov 03 21:00:46.185933 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.185902 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbb5848c7-wcspz_b769fedf-c73d-4aad-90dc-a8a49d0693dc/reload/0.log" Nov 03 21:00:46.212537 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:46.212511 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dbb5848c7-wcspz_b769fedf-c73d-4aad-90dc-a8a49d0693dc/kube-rbac-proxy/0.log" Nov 03 21:00:48.242234 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:48.242205 5954 scope.go:122] "RemoveContainer" containerID="aeadde4b8d916fa6402fdea3ba24b67b57caa30a03fe4c6ef0e63277f5298c7d" Nov 03 21:00:48.760619 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:48.760590 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d455578cf-qdn4g_39df0b28-aa27-4fa6-a126-7c283627fd92/console/0.log" Nov 03 21:00:50.371200 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:50.371089 5954 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8wq99_216dd34c-6ab8-4044-97d2-d2ed537c6153/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:50.682434 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:50.682332 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-chgrg_400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/dns/0.log" Nov 03 21:00:50.751121 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:50.751082 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-chgrg_400dd6c5-8a0e-4d2b-a42e-525ba1b4d0b6/kube-rbac-proxy/0.log" Nov 03 21:00:50.797978 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:50.797952 5954 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-lpczc" Nov 03 21:00:50.946784 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:50.946701 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v2tvg_3371a194-9a9e-4a3a-9af3-48d86152b1dc/dns-node-resolver/0.log" Nov 03 21:00:51.606806 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:51.606777 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-87bj5_c60f9fad-b8c0-4d6e-b755-c2bda3f2d014/node-ca/0.log" Nov 03 21:00:53.103540 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:53.103495 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bzp6h_f29ac0d4-d574-4773-8d82-81f822379713/discovery/0.log" Nov 03 21:00:53.631040 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:53.631005 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-vncsb_6e045ad4-d958-4322-814b-d2c57c8d8cfd/kube-auth-proxy/0.log" Nov 03 21:00:53.801577 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:53.801546 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-mf6wt_ddad7abe-544d-4e16-a193-f7c905fe1e64/istio-proxy/0.log" Nov 03 21:00:53.986503 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:53.986415 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-v6ll6_9142de3d-195d-40ca-92b6-6777196e15d5/istio-proxy/0.log" Nov 03 21:00:54.029764 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:54.029731 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zdxwl_0fb68c68-b69e-4e9b-bc3f-589b465a1ffa/istio-proxy/0.log" Nov 03 21:00:54.102156 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:54.102092 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h4h49_3126a1b2-1b19-441d-912d-7c4b4701b779/payload-pre-processing/0.log" Nov 03 21:00:55.565440 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.565406 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bzp6h_f29ac0d4-d574-4773-8d82-81f822379713/discovery/0.log" Nov 03 21:00:55.599877 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.599849 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-vncsb_6e045ad4-d958-4322-814b-d2c57c8d8cfd/kube-auth-proxy/0.log" Nov 03 21:00:55.686006 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.685975 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-mf6wt_ddad7abe-544d-4e16-a193-f7c905fe1e64/istio-proxy/0.log" Nov 03 21:00:55.784089 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.784056 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-v6ll6_9142de3d-195d-40ca-92b6-6777196e15d5/istio-proxy/0.log" Nov 03 21:00:55.801776 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.801747 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zdxwl_0fb68c68-b69e-4e9b-bc3f-589b465a1ffa/istio-proxy/0.log" Nov 03 21:00:55.845392 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:55.845301 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h4h49_3126a1b2-1b19-441d-912d-7c4b4701b779/payload-pre-processing/0.log" Nov 03 21:00:56.487348 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:56.487320 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8slb6_25f8e8c2-cae1-4e2a-92e0-427d4c4f4186/serve-healthcheck-canary/0.log" Nov 03 21:00:57.292397 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:57.292365 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltbwq_2790512d-386a-41e8-8bb6-31ec8faafbc6/kube-rbac-proxy/0.log" Nov 03 21:00:57.318069 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:57.318033 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltbwq_2790512d-386a-41e8-8bb6-31ec8faafbc6/exporter/0.log" Nov 03 21:00:57.343826 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:57.343801 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ltbwq_2790512d-386a-41e8-8bb6-31ec8faafbc6/extractor/0.log" Nov 03 21:00:59.074195 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.074160 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bzp6h_f29ac0d4-d574-4773-8d82-81f822379713/discovery/0.log" Nov 03 21:00:59.104201 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.104172 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-667774d4b7-vncsb_6e045ad4-d958-4322-814b-d2c57c8d8cfd/kube-auth-proxy/0.log" Nov 03 21:00:59.175216 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.175183 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-mf6wt_ddad7abe-544d-4e16-a193-f7c905fe1e64/istio-proxy/0.log" Nov 03 21:00:59.287761 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.287732 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-v6ll6_9142de3d-195d-40ca-92b6-6777196e15d5/istio-proxy/0.log" Nov 03 21:00:59.368959 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.368934 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zdxwl_0fb68c68-b69e-4e9b-bc3f-589b465a1ffa/istio-proxy/0.log" Nov 03 21:00:59.405618 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.405587 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-h4h49_3126a1b2-1b19-441d-912d-7c4b4701b779/payload-pre-processing/0.log" Nov 03 21:00:59.904237 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.904208 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-64r8n_f46746bf-f978-4b56-8167-0a57befac650/manager/0.log" Nov 03 21:00:59.929254 ip-10-0-135-70 kubenswrapper[5954]: I1103 21:00:59.929222 5954 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-64r8n_f46746bf-f978-4b56-8167-0a57befac650/copy-manifests/0.log" Nov 03 21:01:00.257663 ip-10-0-135-70 kubenswrapper[5954]: E1103 21:01:00.257583 5954 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltbwq" containerName="extractor"